Monday, December 19, 2011

Documentation on handling larger database



Hi Techies,
       I created a documentation for handling larger database. Make use of it & share any other tips which I missed out.

Database server: Buy a server that can handle larger data's

Database engine: We can use either MyISAM or INNODB.

Stick to your basics right:
1)     Create table with proper indexing & primary key

2)     Don't use COUNT * on Innodb tables for every search, do it a few times and/or summary tables, or if you need it for the total # of rows, use SQL_CALC_FOUND_ROWS and SELECT FOUND_ROWS()

3)     Avoid using IN(...) when selecting on indexed fields, It will kill the performance of SELECT query.

4)     Use index to filter rows while searching

5)     Don’t use * in select when you’re not going to retrieve 60% of the columns. Make it specific like (userid, username, doj, lastlogin etc)

6)     Don't Index Everything

7)     A NULL data type can take more room to store than NOT NULL

8)     Use mysql_pconnect instead of mysql_connect in PHP

9)     Don’t use parenthesis in the select query instead use the table name as prefix for the column name.

10)  In the pagination don’t use offset, Just use Limit N. It makes pagination faster.
E.g.: With offset Select * from pages limit 10,20
Without offset Select * from pages where id > 10 limit 20

11)  Use INET_ATON and INET_NTOA for IP addresses

Optimization Tips:
1)     Use OPTIMIZE for each table
a.      E.g. OPTIMIZE Table tablename
                                                              i.      This will work only in MYISAM Engine.

                                                            ii.      While optimizing the entire table will be locked, so it should be done only when the site is offline & maintenance period

2)     To test the server speed, Just execute the following query in mysql
a.      SELECT BENCHMARK(1000000,10+10);

b.      The above command executes the expression 10 + 10 one million times. This will return 0 as result, but the execution time should be noted.
This command as to run different time of the day to check how the server handles the request.

3)     Create index for the table, we should make proper indexing to the fields which we’re going to search for the results.

4)     Before making a select query check it with Explain function & check whether the query is optimized to its best. This will be very useful when we use for complex queries.

5)     Use stored procedures to avoid bandwidth wastage

6)     Partition your database/table

7)     Delete data that you don’t need. Tables will grow, so monitor table sizes regularly.

Secure the Data’s:
While inserting or fetching data’s from the website, we need to be careful. We should follow the steps mentioned below
1)     Use mysql_real_escape_string() for the datas that has user interaction like Inserting fields & query strings

2)     Use POST method as much as possible

3)     Filter each inputs to make it secure. Use strip_tags().

4)     When you're getting the values from query string, use (int) function for getting integer values & if you want to get a string or characters use substring to get the values to a certain limit (For example: An hacker can pass ascii code to the query string & can access the database. So, we can limit upto 10 characters in the query string. If we do so, sql injection will be prevented).

5)     If you're using AJAX, pass the values by post method.

6)     Don't allow php files & other executable files in the file upload option. If you do so, the hackers can write their script in php file & upload it to your server & can access any data’s that they want.

7)     In production path don't print the mysql error code or sql query. If any error displayed then the hackers can find out the table structure for the website & can use it accordingly.

Monday, August 2, 2010

Database Class using PHP

Hi PHP Developers,
I wrote a Database utility class, which is used mostly in any projects. If you've this class, then you don't need to write queries each & every time when a new project comes. You can just copy this file & paste it in your class directory & can call it anytime. The class written below is very simple & it can be understood to the novice also. There will be a way to minimize the functions & statements inside the class too, if so, don't regret to post it in comments.

<?php
/*-----------------------------------------------------------------------
Created By: Sathish Kumar.R
Date: 30 July 2010
E-mail: smart2raise[at]gmail[dot]com
Purpose: Database Manipulation


Functions Available:
db: Constructor (for mysql connect)
query: Executing query(mysql_query)
num_rows: Finding total rows(mysql_num_rows)
fetch_object: Fetch results in Object (mysql_fetch_object)
fetch_array: Fetch results in Array (mysql_fetch_array)
fetch_assoc: Fetch results in Array (mysql_fetch_assoc)
insert: Insert query (Insert into...)
insert_id: Gets last inserted ID (mysql_insert_id)
update Update query (Update ...)
delete Delete query (Delete ...)
countof: Count function in mysql
maxof: Max function in mysql
sumof: Sum function in mysql
avgof: AVG function in mysql
last_query: Displays the query which you executed last
throw_error: If any mysql error occurs & you set $debug = true then it will show the query & the mysql error
-------------------------------------------------------------------------*/
class db{
var $debug = false;
var $query = array();
var $prefix = "";
function db($server,$username,$pwd,$db){
mysql_connect($server,$username,$pwd) or die('Please check your database connection');
mysql_select_db($db);
}
function query($qry){
$this->query[] = $qry;
$res = mysql_query($qry);
if(mysql_error()){
$this->throw_error();
}
return $res;
}
function num_rows($res){
return mysql_num_rows($res);
}
function fetch_object($res){
$fet = mysql_fetch_object($res);
return $fet;
}
function fetch_array($res){
$fet = mysql_fetch_array($res);
return $fet;
}
function fetch_assoc($res){
$fet = mysql_fetch_assoc($res);
return $fet;
}
function insert($val,$table){
$query = 'INSERT INTO '.$table.' (';
foreach ($val AS $key => $value)
$query .= '`'.$key.'`,';
$query = rtrim($query, ',').') VALUES (';
foreach ($val AS $key => $value){
if(get_magic_quotes_gpc())
$query .= '\''.$value.'\',';
else
$query .= '\''.mysql_real_escape_string($value).'\',';
}
$query = rtrim($query, ',').')';
return $this->query($query);
}
function insert_id(){
return mysql_insert_id();
}
function update($val,$table,$con){
if($con!=""){
$where = "where ";
$lastitem = end($con);
foreach ($con AS $key => $value){
if($value!=$lastitem){
if(get_magic_quotes_gpc())
$where .= $key."='".$value."' && ";
else
$where .= $key."='".mysql_real_escape_string($value)."' && ";
}
else{
if(get_magic_quotes_gpc())
$where .= $key."='".$value."'";
else
$where .= $key."='".mysql_real_escape_string($value)."'";
}
}
}
else
{
$where = "";
}
$query = 'update '.$table.' set ';
foreach ($val AS $key => $value){
if(get_magic_quotes_gpc())
$query .= $key."=".'\''.$value.'\',';
else
$query .= '\''.mysql_real_escape_string($value).'\',';
}
$query = rtrim($query, ',')." ".$where;
return $this->query($query);
}
function delete($table,$con){
if($con!=""){
$where = "where ";
$lastitem = end($con);
foreach ($con AS $key => $value){
if($value!=$lastitem){
if(get_magic_quotes_gpc())
$where .= $key."='".$value."' && ";
else
$where .= $key."='".mysql_real_escape_string($value)."' && ";
}
else{
if(get_magic_quotes_gpc())
$where .= $key."='".$value."'";
else
$where .= $key."='".mysql_real_escape_string($value)."'";
}
}
}
else {
$where = "";
}
return $this->query("delete from {$table} {$where}");
}
function countof($col,$table,$con="",$group=""){
if($con!=""){
$where = "where ";
$lastitem = end($con);
foreach ($con AS $key => $value){
if($value!=$lastitem){
if(get_magic_quotes_gpc())
$where .= $key."='".$value."' && ";
else
$where .= $key."='".mysql_real_escape_string($value)."' && ";
}
else{
if(get_magic_quotes_gpc())
$where .= $key."='".$value."'";
else
$where .= $key."='".mysql_real_escape_string($value)."'";
}
}
}
else {
$where = "";
}
if($group!="")
$groupby = "group by ".$group;
else
$groupby = "";
$query = $this->query("select count({$col}) from {$table} {$where} {$groupby}");
$fet = $this->fetch_array($query);
return $fet[0];
}
function maxof($col,$table,$con="",$group=""){
if($con!=""){
$where = "where ";
$lastitem = end($con);
foreach ($con AS $key => $value){
if($value!=$lastitem){
if(get_magic_quotes_gpc())
$where .= $key."='".$value."' && ";
else
$where .= $key."='".mysql_real_escape_string($value)."' && ";
}
else{
if(get_magic_quotes_gpc())
$where .= $key."='".$value."'";
else
$where .= $key."='".mysql_real_escape_string($value)."'";
}
}
}
else {
$where = "";
}
if($group!="")
$groupby = "group by ".$group;
else
$groupby = "";
$query = $this->query("select max({$col}) from {$table} {$where} {$groupby}");
$fet = $this->fetch_array($query);
return $fet[0];
}
function sumof($col,$table,$con="",$group=""){
if($con!=""){
$where = "where ";
$lastitem = end($con);
foreach ($con AS $key => $value){
if($value!=$lastitem){
if(get_magic_quotes_gpc())
$where .= $key."='".$value."' && ";
else
$where .= $key."='".mysql_real_escape_string($value)."' && ";
}
else{
if(get_magic_quotes_gpc())
$where .= $key."='".$value."'";
else
$where .= $key."='".mysql_real_escape_string($value)."'";
}
}
}
else {
$where = "";
}
if($group!="")
$groupby = "group by ".$group;
else
$groupby = "";
$query = $this->query("select sum({$col}) from {$table} {$where} {$groupby}");
$fet = $this->fetch_array($query);
return $fet[0];
}
function avgof($col,$table,$con="",$group=""){
if($con!=""){
$where = "where ";
$lastitem = end($con);
foreach ($con AS $key => $value){
if($value!=$lastitem){
if(get_magic_quotes_gpc())
$where .= $key."='".$value."' && ";
else
$where .= $key."='".mysql_real_escape_string($value)."' && ";
}
else{
if(get_magic_quotes_gpc())
$where .= $key."='".$value."'";
else
$where .= $key."='".mysql_real_escape_string($value)."'";
}
}
}
else {
$where = "";
}
if($group!="")
$groupby = "group by ".$group;
else
$groupby = "";
$query = $this->query("select avg({$col}) from {$table} {$where} {$groupby}");
$fet = $this->fetch_array($query);
return $fet[0];
}
function last_query(){
return end($this->query);
}
function throw_error(){
if($this->debug==true){
$qry = "".end($this->query)."<br>";
}
else{
$qry = "";
}
die("<div style='width:500px; margin:auto; text-align:left; color:red; font-size:12px;border:2px solid #FFD700;vertical-align:middle; line-height:19px;background:#FFFFDD;font-family:verdana;padding:3px;'>".$qry."Mysql Error: ".mysql_error());
}
}
?>

<?php
//Usage of Class:


$hostname = "hostname";
$username = "username";
$password = "";
$database = "dbname";
$db = new db($hostname,$username,$password,$database);
$db->debug = true;
//If it is set to true, then mysql error will shown the query which throws the error.

$qry = $db->query("select columnname from tablename"); //Executing query
$db->num_rows($qry); //Finding total rows

$val = array("a"=>123,"b"=>"sathish","c"=>"kumar");

$a = $db->insert($val,"tablename");
/*Note: $val should be an array, else it will throw error Key in an array should be the column name in the table & value can be any value */

$db->insert_id(); //returns the last inserted ID in the database

$val1 = array("a"=>123,"b"=>"Sathish","c"=>"Kumar");

$db->update($val1,"tablename",array("id"=>2));
/*
Update query same as insert query, you can pass multiple conditions in the 3rd argument i.e array("id"=>2,"a"=>'123')
*/

$db->delete("tablename",array("id"=>4));
//You can pass multiple conditions in 2nd argument


$db->countof("id","tablename",array("a"=>123123));


$db->maxof("id","tablename",array("a"=>123123));


$db->avgof("id","tablename",array("a"=>123123));

/*
for countof(),maxof(),avgof() you can pass additional argument group by


$db->countof("id","tablename",array("a"=>123123),"c");

this will result as "select count(id) where a = '123123' group by c"
*/

$db->last_query() //This will return last executed query
?>

Monday, July 26, 2010

Detect Firebug console using Javascript

Hi Developers,
Nowadays Firebug console is one of the best tool to debug html or javascript or AJAX or css. In the same time it allows any user to see the datas passed in between the pages. It is not a secure method for our website. Gmail has blocked firebug console because of this activities. You can able to login to gmail with firebug console but it will say that "If your firebug console is ON, then gmail will become slow". So you definitely switch off your firebug console. So in the same way, we save our website from firebug.

You can prevent this using javascript.

In your page load call this function
  
<script type='text/javascript'>
function blockfirebug(){
if(window.console.firebug!=""){
document.location.href="firebug_error.html";
return false;
}
}
<script>


<body onload='return blockfirebug()'>