Category Archives: Development

Connect to Sqlite3 using python

The below snippet allows you to create an sqlite3 database and then import data to it.

import sqlite3

#this connects and creates the database.db file, in the same location where the script exists

conn = sqlite3.connect(database.db)
c = conn.cursor()

#execute the create table
c.execute('''CREATE TABLE IF NOT EXISTS mytable
(filename text,
language text, 
year text,
summary text,
rating real)''')

#this is the dataset i will save to database as a python list:

data=['firstfile.mov','en','2018','This is a short summary of the video','3.8']

#get the data in the database
c.execute('INSERT INTO mytable VALUES (?,?,?,?,?)', data)

#close the connection
conn.commit()
conn.close()

htaccess force download file

Using the below part of code inside your htaccess file, you restrict the browser to download the file instead of displaying it in a seperate tab.

Two versions of the code, one for many filetypes and the second just only for pdf:

Various file types:

<FilesMatch "\.(?i:doc|odf|pdf|rtf|txt)$">
  Header set Content-Disposition attachment
</FilesMatch>

Only pdf:

<FilesMatch "\.(?i:doc|odf|pdf|rtf|txt)$">
  Header set Content-Disposition attachment
</FilesMatch>

taken form the drupal comnunity forum

$’\r’: command not found

I usually crate my scripts on my windows everyday computer and upload them to my unix- bash running pc. Many times i get the error :$’\r’: command not found

This is caused by the “CRLF” End  of File for windows, which unix unfortunatelly does not like.

To correct this, open the script using Notepad++ and navigate to Edit>EOL Conversion>Unix(LF)

LF Notepad ++

And then retry the script.

This automatically converts all CRLF to LF. nice hah?

google charts using mysql recordset made easy

General

if your programming skills are limited, then you could use the below code to integrate google charts to your intranet or internet web site.

As i first step  i usually visit the the google charts page and select a chart that matches my desired outcome. Lets say that we have a list of cd collections that we want to process and we need to know graphically how many we have already processed or not. For that i would choose a pie chart.

Mysql Part

Next we need to retrieve the necessary data from the database using the appropriate query.

mysql_query

mysql_queryIn my case i only need to have two columns, one that shows the cd collection visibility type (1=visible, 0= hidden, 3=other status), and the count of each type of visibility.

PHP Part

Next, the php part of the mysql connection:

mysql_select_db($database_localhost, $localhost);
$query_Recordset1 = "select count(*) as count,visible from cdcollection where type=1 and serial=1 group by visible";
$Recordset1 = mysql_query($query_Recordset1, $localhost) or die(mysql_error());
$row_Recordset1 = mysql_fetch_assoc($Recordset1);
$totalRows_Recordset1 = mysql_num_rows($Recordset1);

then we transform the data to an array format:

$data[0] = array('visible','count'); 
 for ($i=1; $i<($totalRows_Recordset1+1); $i++)
 {
 $data[$i] = array(mysql_result($Recordset1, $i-1,'visible'),
 (int) mysql_result($Recordset1, $i-1, 'count'));
 }

Google Charts Script

The below script should be included in the <head> of your document in order to initialize the google charts library.

<script type="text/javascript" src="https://www.google.com/jsapi"></script>

Then right below add the following script part, which basically transform  the data array to json and then it uses it to create the chart.

<script type="text/javascript"> 
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart1);
function drawChart1() {
var data = google.visualization.arrayToDataTable(<?php echo json_encode($data)?>);

 var options = {
 title: 'Collections',
 hAxis: {title: 'Year', titleTextStyle: {color: 'red'}},
 colors: ['#e0440e', '#e6693e', '#ec8f6e', '#f3b49f', '#f6c7b6']
 };


var chart = new google.visualization.PieChart(document.getElementById('chart_div1'));
 chart.draw(data, options);
</script>

I am explaining the red parts of the above code:

  • The json_encode($data) part transform the $data array to json. So in the mysql part if you used some other name for your dataset it should also be changed here.
  • The part right after “var options = {  “ includes all the settings of the chart. This is where you specify the title, the colour style, the fonts etc. Depending on what you need to do adequate documentation exists on the net..
  • Finally, the last red part PieChart is where you tell the api what chart you need it to render. This could be modified to ColumnChart or Histogram or any other form the available options you can find in the google chart development page.

See below the dashboard i have create for my self using three different recordsets.

My Google Charts Dashboard
My Google Charts Dashboard

output_buffering in Wamp causing issues in the insert form

As mentioned earlier this week i changed my development machine to newer vmware win 7. During this, also upgraded from Wamp 2.2 to Wamp 2.5. Fortunatelly, i still keep both environments in parallel since various issues come up.

One of these issues is the one found today in php form (created with the dreamweaver CC wizard) that inserts some values into a mysql table. In my new dev machine i got these errors:

apache erros

The old dev environment worked fine, the prod environment worked fine, the new dev threw errors.

I started by trying to find any compatibility issues in the PHP versions between Wamp Server 2.2 and 2.5, could not locate anything.

The solution came from the Stack Exchange (again). It appears that the php.ini on the new server came with the default value of

output_buffering = 4096

which should be changed to

output_buffering = on

which was the default on the old dev machine. To change the value in Wamp server just go to PHP menu and edit the php.ini to the mentioned value.

wamp.2.5

One alternative suggested by lots of people, which i did not try was to add

ob_start();

at the beggining of the script, which turns on the output buffering feature for the specific script.

published date meta data from mysql timestamp

I am using the timestamp data type in mysql to keep track of when each row has been modified or inserted.

CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP

This saves the date in the database in the form of: 2015-11-24 13:55:10

I need to have a timestamp though, that will be used in the meta tags and will be in the following form :

<meta property="article:published_time" content="2013-09-17T05:59:00+01:00" />

as advised by opengraph guidelines. Googling a lot, and the following are the two candidates on which i ended up to:

date(DATE_ATOM, strtotime($row_getPost['blogpostimestamp']));

which echoes 2015-11-23T15:27:01+00:00

and

date('Y-m-dTH:i:s.uZ',strtotime($row_getPost['blogpostimestamp']));

whch echoes 2015-11-23UTC15:27:01.00

The latter also includes the server timezone.

An alternative to this would be to convert the timestamp, during the query phase, which from what i read is not suggested, and of course did not work for me. Conversion in the php code allows you to use the timestamp with multiple forms in the same page e.g in meta and on footer notifying when this was published and by whom.

 

 

Sitemap & Sitemap Index xml files using php and mysql

For some time now I have been generating sitemap xml files for google submission using Microsoft Excel. Always knew that there would be a way to generate it on the fly/on request and never bother with that again. But how?

Documentation is not very clear because you need to use multiple programming languages, and also because the whole xml validation process seems very sensitive to errors. Since i am not an experienced programmer, stack exchange became my guide and i have finally managed to do it.

If you have multiple pages, you probably need to create intermediate xmlindex pages. These are basically specially marked index files that point to other index files. Warning: Nested xmlindex files are not allowed by Google (even though are at the end followed and indexed).

<?php Header('Content-type: text/xml');

require_once('../Connections/localhost.php'); 

if (!function_exists("GetSQLValueString")) {
function GetSQLValueString($theValue, $theType, $theDefinedValue = "", $theNotDefinedValue = "") 
{
 if (PHP_VERSION < 6) {
 $theValue = get_magic_quotes_gpc() ? stripslashes($theValue) : $theValue;
 }

 $theValue = function_exists("mysql_real_escape_string") ? mysql_real_escape_string($theValue) : mysql_escape_string($theValue);

 switch ($theType) {
 case "text":
 $theValue = ($theValue != "") ? "'" . $theValue . "'" : "NULL";
 break; 
 case "long":
 case "int":
 $theValue = ($theValue != "") ? intval($theValue) : "NULL";
 break;
 case "double":
 $theValue = ($theValue != "") ? doubleval($theValue) : "NULL";
 break;
 case "date":
 $theValue = ($theValue != "") ? "'" . $theValue . "'" : "NULL";
 break;
 case "defined":
 $theValue = ($theValue != "") ? $theDefinedValue : $theNotDefinedValue;
 break;
 }
 return $theValue;
}
}

mysql_select_db($database_localhost, $localhost);
$query_Recordset1 = "select * from sitemaps where `use`='sitemap'";
$Recordset1 = mysql_query($query_Recordset1, $localhost) or die(mysql_error());
$row_Recordset1 = mysql_fetch_assoc($Recordset1);
$totalRows_Recordset1 = mysql_num_rows($Recordset1);
$xml=new SimpleXMLElement('<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
 .'<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
 .'</sitemapindex>'); 
do { $url=$xml->addChild('sitemap');
 $website="http://www.tharsitis.gr/".strtolower(str_replace(' ','-',$row_Recordset1['url']));
 $url->addChild('loc',$website);
 "<br/>";
 //$url->addChild('lastmod','XXXXXXXX'); OPTIONAL
 } while ($row_Recordset1 = mysql_fetch_assoc($Recordset1));
mysql_free_result($Recordset1);
print($xml->asXML());
?>

This is the xml sitemap  index code that worked for me. I created the database recordset using Dreamweaver. It retrieves all the xml index urls from Mysql database.

At the second part of the project i needed to generate the sitemap xml files that the xml index would be point at.

This is the code that worked for me:

<?php Header('Content-type: text/xml');

require_once('../Connections/localhost.php'); 

if (!function_exists("GetSQLValueString")) {
function GetSQLValueString($theValue, $theType, $theDefinedValue = "", $theNotDefinedValue = "") 
{
 if (PHP_VERSION < 6) {
 $theValue = get_magic_quotes_gpc() ? stripslashes($theValue) : $theValue;
 }

 $theValue = function_exists("mysql_real_escape_string") ? mysql_real_escape_string($theValue) : mysql_escape_string($theValue);

 switch ($theType) {
 case "text":
 $theValue = ($theValue != "") ? "'" . $theValue . "'" : "NULL";
 break; 
 case "long":
 case "int":
 $theValue = ($theValue != "") ? intval($theValue) : "NULL";
 break;
 case "double":
 $theValue = ($theValue != "") ? doubleval($theValue) : "NULL";
 break;
 case "date":
 $theValue = ($theValue != "") ? "'" . $theValue . "'" : "NULL";
 break;
 case "defined":
 $theValue = ($theValue != "") ? $theDefinedValue : $theNotDefinedValue;
 break;
 }
 return $theValue;
}
}

$rs1_Recordset1 = "-1";
if (isset($_GET['id'])) {
 $rs_Recordset1 = strtolower(str_replace ( '-', ' ',$_GET['id']));
}

mysql_select_db($database_localhost, $localhost);
$query_Recordset1 = sprintf("SELECT packbrand,packsuuid,packcollection,packsname,folder FROM packs where collectionid=%s and visible='1' order by serial asc",GetSQLValueString($rs_Recordset1, "text"));
$Recordset1 = mysql_query($query_Recordset1, $localhost) or die(mysql_error());
$row_Recordset1 = mysql_fetch_assoc($Recordset1);
$totalRows_Recordset1 = mysql_num_rows($Recordset1);
$xml=new SimpleXMLElement('<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
 .'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">'
 .'</urlset>'); 
do { 
 $brand=$row_Recordset1['packbrand'];
 $collection=$row_Recordset1['packcollection'];
 $url=$xml->addChild('url');
// $image="image:image";
 $website="http://www.tharsitis.gr/".strtolower(str_replace(' ','-',$brand."/".$collection."/".$row_Recordset1['packsuuid']));
 $img="http://www.tharsitis.gr/packagecache/".$row_Recordset1['folder']."/".$row_Recordset1['packsuuid'].".jpg";
 $url->addChild('loc',$website);
 $url->addChild('priority','0.7');
 $image = $url->addChild('image:image', null, 'http://www.google.com/schemas/sitemap-image/1.1');
 $image->addChild('image:loc',$img, 'http://www.google.com/schemas/sitemap-image/1.1');
 $image->addChild('image:caption',$row_Recordset1['packsname'], 'http://www.google.com/schemas/sitemap-image/1.1');
 } while ($row_Recordset1 = mysql_fetch_assoc($Recordset1));
mysql_free_result($Recordset1);
print($xml->asXML());
?>

In both files you can skip the initial part where i build the recordset. This was all done automatically by Dreamweaver. Save the files as php files.
Finally, i needed to modify my .htaccess file so that it translated every php  request matching the above files to xml. This is a requirement to get Google validate the files correctly.

Rewriterule ^xmlindex.xml$ /xml/xmlindex.php [L]
Rewriterule ^p(.*).xml$ /xml/xmlpacks.php?id=$1 [L]

As you can see in the second line of the below htaccess code, I use url variables to generate dynamic xml sitemaps for different sets of urls.  Vuala!

MySQL Max grouping wrongly the data

mysql database
mysql database

I was trying to build a mysql image gallery, and for that reason i wanted to retrieve the largest value, according to a specific column, which would be smaller than a url variable i would retrieve from the user interface. This is the MySql command i used initially and which retrieved wrong results:

select max(idpacks), packuuid FROM table  where idpacks<"60" ;

which returned the following results:

# max(idpacks), packsuuid
'59', '0e1fdf98b90011e48438d067e54001ba'

The problem:

Even though the value 59 in the above result is the correct one, the packuuid does not correspond to the same row. Digging in a bit i found that it actually corresponds to the row which has the smallest idpacks value.  This took me all day to figure out 😀

The cause:

Quoting from web:

MySQL actually permits that an aggregate query returns columns not used in the aggregation (i.e. not listed in GROUP BY clause). It could be considered as flexibility, but in practice this can easily lead to mistakes if a person that designs queries does not understand how they will be executed

So I needed to modify the query.  My SQL query skills are not the best so bare with me. This is the solution I found to get the values i needed to the same row:

SELECT idpacks,packsuuid FROM packs  where idpacks=(select max(idpacks) from packs  where idpacks<"60" );

Which returned the correct values (correct packsuuid for idpacks 59):

# idpacks, packsuuid
'59', '0f6914f0b90011e48438d067e54001ba'

Worked for me!