How do you build a system that can ask people very sensitive questions, for instance about their work, etc; and ensure confidentiality of the answers on a database level, yet allow each person to answer only once (and if necessary edit his answers)?

This is my proposed method.

• The system itself consists of a database, a set of PHP pages, and a configuration file.
• The PHP files identify the user and give him a unique identifier k_u, say, the person’s name, SSN, or similar identity. This is only stored in the current session. Necessary referential constraints can be added to ensure the validity of the identifier, so the user specifies a real name – e.g. through a lookup in a user table.
• The system configuration file contains a system key, k_s, which is a unique, random key, and which is kept secret.
• The database has three tables.
  • Two tables make up the questions. One for each questionnaire, and one for each question connected to the questionnaire. Each questionnaire is given a unique, random key k_q, and each question has an auto-incrementing ID.
  • The third table contains the answers, which every user has specified. The unique key to each answer is given by the question ID, and a cryptographically secure hash h -> H(k_s + k_u + k_q).

By keeping k_s a secret, it is cryptologically impossible to find out the user identity of any given answer in the answers table. And by varying k_q for each questionnaire, it is also impossible to track users over several questionnaires. h will, however, be the same for each given questionnaire and user.

It is thereby possible for the system to post answers from a given user into the tables, and access previous answers; and it is also possible for a user to retrieve answers, filter them, and generate statistics, without ever having access to any identities for the given answers. Even if the system is compromised, it is impossible to learn the identities of any answers without using a brute-force (or table lookup) method.

Does this sound ok?

drop function if exists exceldate;

delimiter //

-- Function that returns an Excel-style or Delphi-style date value
-- from a MySQL date. A date value of 0 represents 1899-12-30.
create function exceldate(p_date date)
    returns int
    sql security invoker
begin
    return to_days(p_date) - 693959;
end //

delimiter ;

select '1899-12-31', exceldate('1899-12-31')     -- should be 1
union
select '2011-10-18', exceldate('2011-10-18');    -- should be 40834

drop function if exists luhn;
drop function if exists luhn_check;

delimiter //

-- Function that calculates a Luhn (mod 10) check digit from a numeric string.
-- The behavior is undefined if the string contains anything else than digits.
-- Assumes that the string does not have a check digit added yet, so it starts
-- with a weight of 2 at the last digit.
create function luhn(p_number varchar(31))
    returns char(1)
    sql security invoker
begin
    declare i, mysum, r, weight int;

    set weight = 2;
    set mysum = 0;
    set i = length(p_number);

    while i > 0 do
        set r = substring(p_number, i, 1) * weight;
        set mysum = mysum + if(r > 9, r - 9, r);
        set i = i - 1;
        set weight = 3 - weight;
    end while;

    return (10 - mysum % 10) % 10;
end //

-- Check if a numeric string has a valid check digit. Does this by cutting off
-- the last digit, recalculating the Luhn check digit, and comparing the strings.
create function luhn_check(p_number varchar(32))
    returns boolean
    sql security invoker
begin
    declare luhn_number varchar(32);

    set luhn_number = substring(p_number, 1, length(p_number) - 1);
    set luhn_number = concat(luhn_number, luhn(luhn_number));

    return luhn_number = p_number;
end //

delimiter ;

I wrote this small script for Windows to compress movies (avi, mkv etc) down to cell phone format. The encoding isn’t great, but the average two-hour movie compresses down to just about ~100 MB, which is really nice. Works fine on Nokia N85. Edit to suit your needs. It relies on ffmpeg, which you can find Windows binary downloads for here and there. Google for it, kid, google for it!

Oh, it also requires cygwin, for the perl interpreter. (Or ActivePerl, I suppose.)

ff-m4v.cmd:

@echo off
set CYGWIN=nodosfilewarning
for %%F in (%*) do (
    echo.
    echo ######## %%F #########
    echo.
    perl %~dpn0.pl %%F
)

ff-m4v.pl:

$in = shift;
$data = `ffmpeg -i '$in' 2>&1`;

$out = $in;
$out =~ s/\.[A-Za-z0-9]+$/\.m4v/;

($x, $y) = $data =~ /Video:.*?(\d+)x(\d+)/i;
$dx = 320 / $x;
$dy = 240 / $y;
$d = $dx < $dy ? $dx : $dy;

$dim = int($d * $x) . 'x' . int($d * $y);

$opt = "ffmpeg -i '$in' -f mp4 -vcodec mpeg4 -b 80000 -r 12 -acodec aac -ar 16000 -ab 48000 -ac 1 -threads 2 -y -s $dim '$out'";
print "$opt\n";
system($opt);

The parameters evaluate to mpeg4 encoding, 80 kbit/s bitrate for video, 12 frames per second; AAC encoding with 48 kbit/s bitrate, mono sound. It compresses the movie down to a maximum of 320x240 pixels, but preserves aspect ratio (effectively becoming 320x196, 320x160 or whatever). If you want to use 2-pass encoding, it's easy; just repeat the system($opt) call with "-pass 1" and "-pass 2" parameters.

Put both in the same folder, make sure ffmpeg.exe is in the system path, and run "ff-m4v *.avi".

I usually sign on to work by starting at least four different programs: Microsoft Lync, Miranda, a SIP telephone, and a little break clock. I used to have a Python script for it, but I thought that installing ActivePython on a Windows machine just to start four programs is a bit of an overkill. So I wrote a CMD file instead. It’s really amazing what you can beat the old CMD interpreter into doing.

startToday.cmd:

@echo off

tasklist /nh > __tasklist.txt

for /f "delims=" %%F in (startToday.files) do (
    findstr /b /i "%%~nxF" __tasklist.txt > nul
    if errorlevel 1 (
        echo starting %%F
        start "" "%%F"
    )
)

del /q /f __tasklist.txt

startToday.files:

C:\Program Files\CounterPath\X-Lite\x-lite.exe
C:\Program Files\Miranda IM\miranda32.exe
C:\Program Files\Microsoft Lync\communicator.exe
C:\Home\stuff\software projects\BreakTime\BreakTime.exe

Voila!

A small batch file I wrote today, to back up MySQL databases, table by table.

So many times have I needed to restore only a specific table, and yet it’s so difficult to dig out the correct table from the whole database backup file (or even worse, server backup file).

This little file will store the whole database/server in individual files, and if you run it on a daily basis, will keep history in “backup.n” directories.

@echo off

rem Enable delayed expansion - otherwise the backup.n folder renaming won't work
setlocal enabledelayedexpansion

rem Change to the script's home folder
%~d0
cd %~p0

rem Rotate previous backup folders
if exist backup.9 rd /s /q backup.9
for /l %%i in (8 -1 1) do (
    set /a j=%%i + 1
    if exist backup.%%i move backup.%%i backup.!j! > nul
)
if exist backup move backup backup.1 > nul

md backup

rem Change these to reflect your MySQL connection parameters
set OPT=-ubackup -pbackup -hlocalhost

rem List all the databases and process them
mysql %OPT% -e"show databases" --skip-column-names -B | findstr /v "information_schema" > _db.txt
for /f %%d in (_db.txt) do call :backupdb %%d

rem Clean up, clean up, everybody wants to clean up
del /f /q _dbt.txt
del /f /q _db.txt

dir backup

goto :eof

:backupdb
    rem List the tables in the current database
    echo %1
    mysql %OPT% -e"show tables" --skip-column-names -B %1 > _dbt.txt
    md backup\work

    rem Dump each table into an SQL file
    echo  - dumping data
    for /f %%t in (_dbt.txt) do mysqldump --opt %OPT% -rbackup\work\%%t.sql %1 %%t

    rem Compress them with 7-zip into a single zip file
    echo  - compressing
    cd backup\work
    ..\..\7z -y a ..\%1.zip * > nul

    rem Clean up
    cd ..
    rd /s /q work
    cd ..
    echo.

    goto :eof

Download 7-zip, extract 7z.exe and 7z.dll and put in the same folder as the “backup.cmd” file above, and you should be good to go. Change the -u and -p parameters in the “set OPT” statement to reflect the login settings for your database server.

/**
 *  NF_Persistence
 *
 *  Agent Smith: "Why, Mr. Anderson, why? Why do you persist?"
 *  Neo:         "Because I choose to."
 *
 *  PHP Version 5
 *
 *  @category   NiftyFramework
 *  @package    NiftyFramework
 *  @subpackage Database
 *  @author     Mats Gefvert

 *  @license    http://www.sun.com/cddl/ Common Development and Distribution License
 */

class NF_PersistenceRelationMap
{
    ....

I make myself laugh sometimes. Is that a good sign? :)

I’ve found a new great blog, Gustavo Duarte’s blog, where he talks about hardware, operating system kernerls, and other fun items.

I liked his comparison of how fast different types of memory is.

The first thing that jumps out is how absurdly fast our processors are. Most simple instructions on the Core 2 take one clock cycle to execute, hence a third of a nanosecond at 3.0Ghz. For reference, light only travels ~4 inches (10 cm) in the time taken by a clock cycle…

Normally when the CPU needs to touch the contents of a memory region they must either be in the L1/L2 caches already or be brought in from the main system memory … To put this into perspective, reading from L1 cache is like grabbing a piece of paper from your desk (3 seconds), L2 cache is picking up a book from a nearby shelf (14 seconds), and main system memory is taking a 4-minute walk down the hall to buy a Twix bar…

[...] Even main memory is blazing fast compared to hard drives. Keeping with the office analogy, waiting for a hard drive seek is like leaving the building to roam the earth for one year and three months. This is why so many workloads are dominated by disk I/O and why database performance can drive off a cliff once the in-memory buffers are exhausted …

Subscribed. :)