I have developed an application which uses SQL Server as a data store and was looking for a way to install it with my appliation in a seamless fashion. With a lot of trial and error and a couple of hours trawling the internet for a decent source for this kind of info, I found a very interesting page on Microsoft's website.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsse/html/EmSQLExCustApp.asp

Finally, I decided to use the command prompt to  launch the installer.

SQLEXPR.EXE -q /norebootchk /qb reboot=ReallySuppress addlocal=all instancename=SQLCatalog SCCCHECKLEVEL=IncompatibleComponents:1;MDAC25Version:0 ERRORREPORTING=1 SQLAUTOSTART=1 SAPWD=SQLPassword SECURITYMODE=SQL DISABLENETWORKPROTOCOLS=0

Then I adapted an NSIS script I had (see below). All you need to do then is call the function UpdateMSDE from your NSIS script.

; English
!define URL_MSDE "http://www.microsoft.com/downloads/info.aspx?na=46&p=8&SrcDisplayLang=en&SrcCategoryId=&SrcFamilyId=57856CDD-DA9B-4AD0-9A8A-F193AE8410AD&u=http%3a%2f%2fdownload.microsoft.com%2fdownload%2ff%2fd%2f4%2ffd407f12-0845-4d51-b084-779860d57ba5%2fSQLEXPR.EXE&oRef=http%3a%2f%2fwww.microsoft.com%2fsql%2feditions%2fexpress%2fdefault.mspx"

Function TestMSDE

        ;MessageBox MB_OK "Checking $1   $2"
 ReadRegStr $0 HKLM \
        "$1" "$2"

        IfErrors SQLServerNotFound SQLServerFound

 SQLServerFound:
                ;MessageBox MB_OK "Checking $3   $4"
  ReadRegStr $5 HKLM  $3 $4
                IfErrors SPNotFound SPFound

 SPFound:
  ;Check the first digit of the version; must be 8
  StrCpy $0 $5
  StrCpy $1 $0 1
  StrCmp $1 "8" SQLServer2000Found SQLServerVersionError

 SQLServer2000Found:
                Push 1
  Goto ExitCheckMinSQLVersion

 SQLServerVersionError:
  ;MessageBox MB_OK|MB_ICONEXCLAMATION  "This product requires a minimum SQLServer version of 8; detected version $0. Setup will install SQLExpress 2005."
  Push 0
  Goto ExitCheckMinSQLVersion

 SQLServerNotFound:
  ;MessageBox MB_OK|MB_ICONEXCLAMATION  "SQLServer was not detected; this is required for installation. Setup will install SQLExpress 2005."
                Push 0
  Goto ExitCheckMinSQLVersion

 SPNotFound:
  ;MessageBox MB_OK|MB_ICONEXCLAMATION  "SQLServer version $0 was detected. SQLServer version 8 (or later) is required for installation. Setup will install SQLExpress 2005."
  Push 0
  Goto ExitCheckMinSQLVersion

 ExitCheckMinSQLVersion:
 
FunctionEnd

Function ExistsMSDE

   ; search for SQL server
   StrCpy $1 "SOFTWARE\Microsoft\MSSQLServer\MSSQLServer\CurrentVersion"
   StrCpy $2 "CurrentVersion"
   StrCpy $3 "SOFTWARE\Microsoft\MSSQLServer\MSSQLServer\CurrentVersion"
   StrCpy $4 "CSDVersion"
   Call TestMSDE
  
   pop $1
   IntCmp $1 1 Found
  
   ; search for SQLExpress 2005 instance
   StrCpy $1 "SOFTWARE\Microsoft\Microsoft SQL Server\$SQLCatalog\MSSQLServer\CurrentVersion"
   StrCpy $2 "CurrentVersion"
   StrCpy $3 "SOFTWARE\Microsoft\Microsoft SQL Server\$SQLCatalog\MSSQLServer\CurrentVersion"
   StrCpy $4 "CSDVersion"
   Call TestMSDE
  
   pop $1
   IntCmp $1 1 Found
  
   ;jump to not found which will then start the installer
   ;IfFileExists 'C:\MSDERelA\Setup.exe' NotFound
  
   ; let's try to download SQLExpress 2005
   MessageBox MB_YESNO|MB_ICONEXCLAMATION  "SQLExpress 2005 not found. Do you want me to download installer from ${URL_MSDE}?" IDYES true IDNO false

true:
   nsisdl::download /TIMEOUT=30000 "${URL_MSDE}" "$PLUGINSDIR\SQLEXPR.EXE"
   Pop $0
   StrCmp $0 "success" Downloaded
   MessageBox MB_OK "Download failed: $0"
  
false:
   MessageBox MB_OK "Failed to detect MSDE or SQLExpress on this machine. The installer will continue but the application will fail to start. You will have to manually install the database engine and run the database creation scripts."
   Goto NotFound
  
Downloaded:
   ; run installer
   ExecWait "$PLUGINSDIR\SQLEXPR.EXE -q /norebootchk /qb reboot=ReallySuppress addlocal=all instancename=$SQLCatalog SCCCHECKLEVEL=IncompatibleComponents:1;MDAC25Version:0 ERRORREPORTING=1 SQLAUTOSTART=1 SAPWD=$SQLPassword SECURITYMODE=SQL DISABLENETWORKPROTOCOLS=0"
   ;ExecWait 'net start mssql$$$SQLCatalog'
   Sleep 10000 ; wait for a 10 seconds
   Goto Found
  
NotFound:
   Push 0
  
   Goto Exit
  
Found:
   Push 1
  
Exit:
  
FunctionEnd

Function UpdateMSDE
   Call ExistsMSDE
   pop $1
  
   IntCmp $1 1 Found
 
Found: 
Exit:

FunctionEnd

Yes I know. The script needs a bit of cleaning. But I trust you can do it.