PAR::Tutorial(3pm)

On Deploying Perl Applications

 % sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101"
 Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1.
 BEGIN failed--compilation aborted at sshnuke.pl line 1.

•

Q: "Help! I can't run your program!"

•

A1: Install Perl & "perl -MCPAN -e'install(...)'"

•

How do we know which modules are needed?

•

New versions of CPAN modules may break "sshnuke.pl"

•

A2: Install Perl & "tar zxf my_perllib.tgz"

•

Possibly overwriting existing modules; not cross-platform at all

•

A3: Use the executable generated by "perlcc sshnuke.pl"

•

Impossible to debug; "perlcc" usually does not work anyway

PAR, the Perl Archive Toolkit

•

Do what JAR (Java Archive) does for Perl

•

Aggregates modules, scripts and other files into a Zip file

•

Easy to generate, update and extract

•

Version consistency: solves forward-compatibility problems

•

Developed by community: "[email protected]"

•

PAR files can be packed into self-contained scripts

•

Automatically scans perl script for dependencies

•

Bundles all necessary 3rd-party modules with it

•

Requires only core Perl to run on the target machine

•

PAR also comes with "pp", the Perl Packager:

 % pp -o sshnuke.exe sshnuke.pl # stand-alone executable!

Simple Packaging

•

PAR files are just Zip files with modules in it

•

Any Zip tools can generate them:

 % zip foo.par Hello.pm World.pm        # pack two modules
 % zip -r bar.par lib/          # grab all modules in lib/

•

To load modules from PAR files:

 use PAR;
 use lib "foo.par";             # the .par part is optional
 use Hello;

•

This also works:

 use PAR "/home/mylibs/*.par";  # put all of them into @INC
 use Hello;

PAR Loaders

•

Use "par.pl" to run files inside a PAR archive:

 % par.pl foo.par               # looks for 'main.pl' by default
 % par.pl foo.par test.pl       # runs script/test.pl in foo.par

•

Same thing, with the stand-alone "parl" or "parl.exe":

 % parl foo.par                 # no perl or PAR.pm needed!
 % parl foo.par test.pl         # ditto

•

The PAR loader can prepend itself to a PAR file:

•

"-b" bundles non-core modules needed by "PAR.pm":

 % par.pl -b -O./foo.pl foo.par # self-contained script

•

"-B" bundles core modules in addition to "-b":

 % parl -B -O./foo.exe foo.par  # self-contained binary

Dependency Scanning

•

Recursively scan dependencies with "scandeps.pl":

 % scandeps.pl sshnuke.pl
 # Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN
 'Crypt::SSLeay'       => '0', #  X   #
 'Net::HTTP'           => '0', #      #
 'Crypt::SSLeay::X509' => '0', # S    # Crypt::SSLeay
 'Net::HTTP::Methods'  => '0', # S    # Net::HTTP
 'Compress::Zlib'      => '0', #  X   # Net::HTTP::Methods

•

Scan an one-liner, list all involved files:

 % scandeps.pl -V -e "use Dynaloader;"
 ...
 # auto/DynaLoader/dl_findfile.al [autoload]
 # auto/DynaLoader/extralibs.ld [autoload]
 # auto/File/Glob/Glob.bs [data]
 # auto/File/Glob/Glob.so [shared]
 ...

Perl Packager: "pp"

•

Combines scanning, zipping and loader-embedding:

 % pp -o out.exe src.pl         # self-contained .exe
 % out.exe                      # runs anywhere on the same OS

•

Bundle additional modules:

 % pp -o out.exe -M CGI src.pl  # pack CGI + its dependencies, too

•

Pack one-liners:

 % pp -o out.exe -e 'print "Hi!"'   # turns one-liner into executable

•

Generate PAR files instead of executables:

 % pp -p src.pl                 # makes 'source.par'
 % pp -B -p src.pl              # include core modules

How it works

•

Command-line options are almost identical to "perlcc"'s

•

Also supports "gcc"-style long options:

 % pp --gui --verbose --output=out.exe src.pl

•

Small initial overhead; no runtime overhead

•

Dependencies are POD-stripped before packing

•

Loads modules directly into memory on demand

•

Shared libraries (DLLs) are extracted with File::Temp

•

Works on Perl 5.6.0 or above

•

Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64...

Aggregating multiple programs

•

A common question:

 > I have used pp to make several standalone applications which work
 > great, the only problem is that for each executable that I make, I am
 > assuming the parl.exe is somehow bundled into the resulting exe.

•

The obvious workaround:

 You can ship parl.exe by itself, along with .par files built
 by "pp -p", and run those PAR files by associating them to parl.exe.

•

On platforms that have "ln", there is a better solution:

 % pp --output=a.out a.pl b.pl  # two scripts in one!
 % ln a.out b.out               # symlink also works
 % ./a.out                      # runs a.pl
 % ./b.out                      # runs b.pl

Cross-platform Packages

•

Of course, there is no cross-platform binary format

•

Pure-perl PAR packages are cross-platform by default

•

However, XS modules are specific to Perl version and platform

•

Multiple versions of a XS module can co-exist in a PAR file

•

Suppose we need "out.par" on both Win32 and Finix:

 C:\> pp --multiarch --output=out.par src.pl
 ...copy src.pl and out.par to a Finix machine...
 % pp --multiarch --output=out.par src.pl

•

Now it works on both platforms:

 % parl out.par                 # runs src.pl
 % perl -MPAR=out.par -e '...'  # uses modules inside out.par

The Anatomy of a PAR file

•

Modules can reside in several directories:

 /                      # casual packaging only
 /lib/                  # standard location
 /arch/                 # for creating from blib/ 
 /i386-freebsd/         # i.e. $Config{archname}
 /5.8.0/                # i.e. Perl version number
 /5.8.0/i386-freebsd/   # combination of the two above

•

Scripts are stored in one of the two locations:

 /                      # casual packaging only
 /script/               # standard location

•

Shared libraries may be architecture- or perl-version-specific:

 /shlib/(5.8.0/)?(i386-freebsd/)?

•

PAR files may recursively contain other PAR files:

 /par/(5.8.0/)?(i386-freebsd/)?

Special files

•

MANIFEST

•

Index of all files inside PAR

•

Can be parsed with "ExtUtils::Manifest"

•

META.yml

•

Dependency, license, runtime options

•

Can be parsed with "YAML"

•

SIGNATURE

•

OpenPGP-signed digital signature

•

Can be parsed and verified with "Module::Signature"

Advantages over perlcc, PerlApp and Perl2exe

•

This is not meant to be a flame

•

All three maintainers have contributed to PAR directly; I'm grateful

•

perlcc

•

"The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged." (from perldoc perlcc)

•

Guaranteed to not work is more like it

•

PerlApp / Perl2exe

•

Expensive: Need to pay for each upgrade

•

Non-portable: Only available for limited platforms

•

Proprietary: Cannot extend its features or fix bugs

•

Obfuscated: Vendor and black-hats can see your code, but you can't

•

Inflexible: Does not work with existing Perl installations

MANIFEST: Best viewed with Mozilla

•

The URL of "MANIFEST" inside "/home/autrijus/foo.par":

 jar:file:///home/autrijus/foo.par!/MANIFEST

•

Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:

•

No needed to unzip anything; just click on files to view them

META.yml: Metadata galore

•

Static, machine-readable distribution metadata

•

Supported by "Module::Build", "ExtUtils::MakeMaker", "Module::Install"

•

A typical "pp"-generated "META.yml" looks like this:

 build_requires: {}
 conflicts: {}
 dist_name: out.par
 distribution_type: par
 dynamic_config: 0
 generated_by: 'Perl Packager version 0.03'
 license: unknown
 par:
   clean: 0
   signature: ''
   verbatim: 0
   version: 0.68

•

The "par:" settings controls its runtime behavior

SIGNATURE: Signing and verifying packages

•

OpenPGP clear-signed manifest with SHA1 digests

•

Supported by "Module::Signature", "CPANPLUS" and "Module::Build"

•

A typical "SIGNATURE" looks like this:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS
 ...
 -----BEGIN PGP SIGNATURE-----
 ...
 -----END PGP SIGNATURE-----

•

Use "pp" and "cpansign" to work with signatures:

 % pp -s -o foo.par bar.pl      # make and sign foo.par from bar.pl
 % cpansign -s foo.par  # sign this PAR file
 % cpansign -v foo.par  # verify this PAR file

Perl Servlets with Apache::PAR

•

Framework for self-contained Web applications

•

Similar to Java's "Web Application Archive" (WAR) files

•

Works with mod_perl 1.x or 2.x

•

A complete web application inside a ".par" file

•

Apache configuration, static files, Perl modules...

•

Supports Static, Registry and PerlRun handlers

•

Can also load all PARs under a directory

•

One additional special file: "web.conf"

 Alias /myapp/cgi-perl/ ##PARFILE##/
 <Location /myapp/cgi-perl>
     Options +ExecCGI
     SetHandler perl-script
     PerlHandler Apache::PAR::Registry
 </Location>

Hon Dah, A-par-che!

•

First, make a "hondah.par" from an one-liner:

 # use the "web.conf" from the previous slide
 % pp -p -o hondah.par -e 'print "Hon Dah!\n"' \
      --add web.conf
 % chmod a+x hondah.par

•

Add this to "httpd.conf", then restart apache:

 <IfDefine MODPERL2>
 PerlModule Apache2
 </IfDefine>
 PerlAddVar PARInclude /home/autrijus/hondah.par
 PerlModule Apache::PAR

•

Test it out:

 % GET http://localhost/myapp/cgi-perl/main.pl
 Hon Dah!

•

Instant one-liner web application that works!

On-demand library fetching

•

With LWP installed, your can use remote PAR files:

 use PAR;
 use lib 'http://aut.dyndns.org/par/DBI-latest.par';
 use DBI;    # always up to date!

•

Modules are now cached under $ENV{PAR_GLOBAL_TEMP}

•

Auto-updates with "LWP::Simple::mirror"

•

Download only if modified

•

Safe for offline use after the first time

•

May use "SIGNATURE" to prevent DNS-spoofing

•

Makes large-scale deployment a breeze

•

Upgrades from a central location

•

No installers needed

Code Obfuscation

•

Also known as source-hiding techniques

•

It is not encryption

•

Offered by PerlApp, Perl2Exe, Stunnix...

•

Usually easy to defeat

•

Take optree dump from memory, feed to "B::Deparse"

•

If you just want to stop a casual "grep", "deflate" already works

•

PAR now supports pluggable input filters with "pp -f"

•

Bundled examples: Bleach, PodStrip and PatchContent

•

True encryption using "Crypt::*"

•

Or even _product activation_ over the internet

•

Alternatively, just keep core logic in your server and use RPC

Accessing packed files

•

To get the host archive from a packed program:

 my $zip = PAR::par_handle($0); # an Archive::Zip object
 my $content = $zip->contents('MANIFEST');

•

Same thing, but with read_file():

 my $content = PAR::read_file('MANIFEST');

•

Loaded PAR files are stored in %PAR::LibCache:

 use PAR '/home/mylibs/*.par';
 while (my ($filename, $zip) = each %PAR::LibCache) {
     print "[$filename - MANIFEST]\n";
     print $zip->contents('MANIFEST');
 }

Packing GUI applications

•

GUI toolkits often need to link with shared libraries:

 # search for libncurses under library paths and pack it
 % pp -l ncurses curses_app.pl  # same for Tk, Wx, Gtk, Qt...

•

Use "pp --gui" on Win32 to eliminate the console window:

 # pack 'src.pl' into a console-less 'out.exe' (Win32 only)
 % pp --gui -o out.exe src.pl

•

"Can't locate Foo/Widget/Bar.pm in @INC"?

•

Some toolkits (notably Tk) autoloads modules without "use" or "require"

•

Hence "pp" and "Module::ScanDeps" may fail to detect them

•

Tk problems mostly fixed by now, but other toolkits may still break

•

You can work around it with "pp -M" or an explicit "require"

•

Or better, send a short test-case to "[email protected]" so we can fix it

Precompiled CPAN distributions

•

Installing XS extensions from CPAN was difficult

•

Some platforms do not come with a compiler (Win32, MacOSX...)

•

Some headers or libraries may be missing

•

PAR.pm itself used to suffer from both problems

•

...but not anymore -- "Module::Install" to the rescue!

 # same old Makefile.PL, with a few changes
 use inc::Module::Install;      # was "use ExtUtils::MakeMaker;"
 WriteMakefile( ... );          # same as the original
 check_nmake();                 # make sure the user have nmake
 par_base('AUTRIJUS');          # your CPAN ID or a URL
 fetch_par() unless can_cc();   # use precompiled PAR only if necessary

•

Users will not notice anything, except now it works

•

Of course, you still need to type "make par" and upload the precompiled package

•

PAR users can also install it directly with "parl -i"

Thank you!

•

Additional resources

•

Mailing list: "[email protected]"

•

Subscribe: Send a blank email to "[email protected]"

•

List archive: <http://nntp.x.perl.org/group/perl.par>

•

PAR::Intro: <http://search.cpan.org/dist/PAR/lib/PAR/Intro.pod>

•

Apache::PAR: <http://search.cpan.org/dist/Apache-PAR/>

•

Module::Install: <http://search.cpan.org/dist/Module-Install/>

•

Any questions?

Overview of PAR.pm's Implementation

•

Here begins the scary part

•

Grues, Dragons and Jabberwocks abound...

•

You are going to learn weird things about Perl internals

•

PAR invokes four areas of Perl arcana:

•

@INC code references

•

On-the-fly source filtering

•

Overriding DynaLoader::bootstrap() to handle XS modules

•

Making self-bootstrapping binary executables

•

The first two only works on 5.6 or later

•

DynaLoader and %INC are there since Perl 5 was born

•

PAR currently needs 5.6, but a 5.005 port is possible

Code References in @INC

•

On 1999-07-19, Ken Fox submitted a patch to P5P

•

To _enable using remote modules_ by putting hooks in @INC

•

It's accepted to come in Perl 5.6, but undocumented until 5.8

•

Type "perldoc -f require" to read the nitty-gritty details

•

Coderefs in @INC may return a fh, or undef to 'pass':

 push @INC, sub {
     my ($coderef, $filename) = @_;  # $coderef is \&my_sub
     open my $fh, "-|", "wget ftp://example.com/$filename";
     return $fh;        # using remote modules, indeed!
 };

•

Perl 5.8 let you open a file handle to a string, so we just use that:

        open my $fh, "<", \($zip->memberNamed($filename)->contents);
        return $fh;

•

But Perl 5.6 does not have that, and I don't want to use temp files...

Source Filtering without Filter::* Modules

•

... Undocumented features to the rescue!

•

It turns out that @INC hooks can return two values

•

The first is still the file handle

•

The second is a code reference for line-by-line source filtering!

•

This is how "Acme::use::strict::with::pride" works:

 # Force all modules used to use strict and warnings
 open my $fh, "<", $filename or return;
 my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n");
 return ($fh, sub {
     return 0 unless @lines;    
     push @lines, $_; $_ = shift @lines; return length $_;
 });

Source Filtering without Filter::* Modules (cont.)

•

But we don't really have a filehandle for anything

•

Another undocumented feature saves the day!

•

We can actually omit the first return value altogether:

 # Return all contents line-by-line from the file inside PAR
 my @lines = split(
     /(?<=\n)/,
     $zip->memberNamed($filename)->contents
 );
 return (sub {
     $_ = shift(@lines);
     return length $_;
 });

Overriding DynaLoader::bootstrap

•

XS modules have dynamically loaded libraries

•

They cannot be loaded as part of a zip file, so we extract them out

•

Must intercept DynaLoader's library-finding process

•

Module names are passed to "bootstrap" for XS loading

•

During the process, it calls "dl_findfile" to locate the file

•

So we install pre-hooks around both functions

•

Our "_bootstrap" just checks if the library is in PARs

•

If yes, extract it to a "File::Temp" temp file

•

The file will be automatically cleaned up when the program ends

•

It then pass the arguments to the original "bootstrap"

•

Finally, our "dl_findfile" intercepts known filenames and return it

Anatomy of a Self-Contained PAR executable

•

The par script ($0) itself

•

May be in plain-text or native executable format

•

Any number of embedded files

•

Typically used to bootstrap PAR's various dependencies

•

Each section begins with the magic string "FILE"

•

Length of filename in pack('N') format and the filename (auto/.../)

•

File length in pack('N') and the file's content (not compressed)

•

One PAR file

•

Just a regular zip file with the magic string "PK\003\004"

•

Ending section

•

A pack('N') number of the total length of FILE and PAR sections

•

Finally, there must be a 8-bytes magic string: "\012PAR.pm\012"

Self-Bootstrapping Tricks

•

All we can expect is a working perl interpreter

•

The self-contained script *must not* use any modules at all

•

But to process PAR files, we need XS modules like Compress::Zlib

•

Answer: bundle all modules + libraries used by PAR.pm

•

That's what the "FILE" section in the previous slide is for

•

Load modules to memory, and write object files to disk

•

Then use a local @INC hook to load them on demand

•

Minimizing the amount of temporary files

•

First, try to load PerlIO::scalar and File::Temp

•

Set up an END hook to unlink all temp files up to this point

•

Load other bundled files, and look in the compressed PAR section

•

This can be much easier with a pure-perl inflate(); patches welcome!

Thank you (again)!

•

Any questions, please?