--- name: "ExtUtils::xsbuilder.osc2002" section: "3pm" platform: "linux" package: "perl-extutils-xsbuilder" version: "0.28-11" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/ExtUtils::xsbuilder.osc2002" --- # ExtUtils::xsbuilder.osc2002(3pm) ## XSBuilder - Autogenerating XS-glue Code O'Reilly OpenSource Convention 2002 Gerald Richter ecos gmbh http://www.ecos.de ## XSBuilder - What is it? ### It's purpose is to automaticly generate a Perl interface to C Code ### Solves the problem, that the Perl interface is not always uptodate with the C interface ### Saves a lot of Copy&Paste work ### Systematical changes have to be done only once For example changes in the memory management of strings. ### Is part of mod\_perl 2.0 build system Most code is developed by Doug MacEachern. Additionaly I have - abstracted the code from mod\_perl so it's useable for any C code - added new features like callbacks and the ability to parse comments - Replaced C::Scan with a Parse::RecDescent to be platform and compiler independend Goal is to replace the current mod\_perl XS generation code with XSBuilder ### Inline versus XSBuilder - Inline: embed C-Code into Perl - XSBuilder: Create interface for existing C-libraries/applicationen ## XSBuilder - What does it do? ### Create Perl functions/methods for every C function The function can be assigned to different packages, also automaticly by inspecting the first parameter ### Create a Perl class for every C structure Every element of structure becomes a Perl method to get/set it's value. The object can be either a scalar reference (used by mod\_perl) or a reference to a hash (use by Embperl), which allows to store extra data by the Perl code into this hash. ### Create glue code to handle callbacks There several sorts of callback, not all are implemented right now ### Create Perl constant subs Coming soon... ## XSBuilder - How does it work? ### Parse the C header files Extract - Functions, their arguments and return types - Structures and it's members - Constants - Callbacks and create four tables which contains the results ### Create the XS code Input is - The source tables - Mapfiles which contains the mapping from C to Perl - Addtional C and Perl code that can be used to customize the interface Output is - The XS files (one form every generated class) - Makefile.PL for every class - pm files ## Parse the source ### Create your own ParseSource class and override methods... ``` package Apache::DAV::ParseSource; use strict; use vars qw{@ISA $VERSION} ; use ExtUtils::XSBuilder::ParseSource ; @ISA = ('ExtUtils::XSBuilder::ParseSource') ; $VERSION = '0.01'; my $dav_dir = 'C:\perl\msrc\cvs\mod_dav' ; my $ap_dir = 'c:\programme\apache group\apache' ; # ============================================================================ sub find_includes { my $self = shift; return $self->{includes} if $self->{includes}; my @includes = ("$ap_dir/include/ap_alloc.h", "$dav_dir/mod_dav.h") ; return $self->{includes} = $self -> sort_includes (\@includes) ; } # ============================================================================ sub package { 'Apache::DAV' } # ============================================================================ sub preprocess { my $self = shift ; $_[0] =~ s/(?:API_EXPORT)(?:_NONSTD)?\s*\(\s*(.*?)\s*\)/$1/g ; } 1; ``` ### ...run it ``` use FindBin ; use lib ($FindBin::Bin) ; require ParseSource ; Apache::DAV::ParseSource -> run ; ``` ### ...and you get ``` C:\perl\msrc\davint>perl xsbuilder\source_scan.pl Will use mod_dav in C:\perl\msrc\cvs\mod_dav Will use Apache in c:\programme\apache group\apache Initialize parser scan c:\programme\apache group\apache/include/ap_alloc.h ... constant: APACHE_ALLOC_H func: ap_init_alloc func: ap_cleanup_alloc func: ap_make_sub_pool func: ap_destroy_pool constant: ap_pool_join func: ap_pool_join func: ap_find_pool func: ap_pool_is_ancestor func: ap_clear_pool func: ap_cleanup_for_exec func: ap_palloc func: ap_pcalloc func: ap_pstrdup func: ap_pstrndup func: ap_pstrcat func: ap_pvsprintf valuefield: ap_pool * : pool valuefield: int : elt_size valuefield: int : nelts valuefield: int : nalloc valuefield: char * : elts struct: (type=array_header) ... ``` ### The result is stored in four tables - xsbuilder/tables/Apache/DAV/FuntionTable.pm Contains all function, it arguments and comments - xsbuilder/tables/Apache/DAV/ConstantTable.pm Contains all constants - xsbuilder/tables/Apache/DAV/StructureTable.pm Contains all structures, it's members and their comments - xsbuilder/tables/Apache/DAV/CallbackTable.pm Contains all callback function definitions ## Create the map files ### Mapfiles are used to tell XSBuilder how C datatypes, structures and function aruments should be mapped into Perl ones. ### Create your own WrapXS class and override methods ``` package Apache::DAV::WrapXS ; use strict; use vars qw{@ISA $VERSION} ; use ExtUtils::XSBuilder::WrapXS ; @ISA = ('ExtUtils::XSBuilder::WrapXS') ; $VERSION = '0.01'; # ============================================================================ sub new_parsesource { [ Apache::DAV::ParseSource->new ] } # ============================================================================ sub my_xs_prefix { 'davxs_' } # ============================================================================ sub h_filename_prefix { 'moddav_xs_' } # ============================================================================ sub xs_includes { my $self = shift ; my $i = $self -> SUPER::xs_includes ; my @i = grep (!/ap_alloc/, @$i) ; return \@i ; } ``` ### XSBuilder can create/update initial maps for you ``` use FindBin ; use lib ($FindBin::Bin) ; require ParseSource ; require WrapXS ; Apache::DAV::WrapXS->checkmaps (' '); ``` ### run it ``` C:\perl\msrc\davint>perl xsbuilder\xs_check.pl Will use mod_dav in C:\perl\msrc\cvs\mod_dav Will use Apache in c:\programme\apache group\apache Parse xsbuilder\maps/_types.map... WARNING: No *_function.map file found in xsbuilder\maps WARNING: No *_callback.map file found in xsbuilder\maps WARNING: No *_structure.map file found in xsbuilder\maps Write xsbuilder\maps/new_function.map... Write xsbuilder\maps/new_callback.map... Write xsbuilder\maps/new_structure.map... Write xsbuilder\maps/new_type.map... ``` ### Now we have four map files - new\_types.map Contains the mapping from C type to Perl classes - new\_functions.map Contains the mapping form C functions to Perl functions. Can be used to reorder arguments, tell XSBuilder which arguments are actualy return values and in which Perl package the function will be created. - new\_structures.map Contains the mapping from C structures to Perl classes and defines for which members a access methods should be created. You can also specify if you want a `"new"` method for the class. - new\_callbacks.map Contains the mapping form C callback functions to Perl callback functions. Can be used to reorder arguments, tell XSBuilder which arguments are actualy return values and in which Perl package the function will be created. It's a good idea to rename the prefix from `"new_"` to something unique, here we use `"dav"` Everytime you rerun checkmaps, XSBuilder will create new\_\* files with the items that are not already part of the other maps. ### Next step is to customize the maps... ## type map ### autogenerated dav\_type.map ``` DIR | FILE | HANDLE | array_header | dav_buffer | dav_dyn_context | dav_dyn_hooks | dav_dyn_module | dav_dyn_provider | dav_error | dav_get_props_result | dav_hooks_liveprop | dav_hooks_locks | dav_hooks_repository | dav_if_header | dav_if_state_type | ... ``` ### Add Perl classes ``` struct array_header | Apache::Array struct dav_buffer | struct dav_datum | Apache::DAV::Datum struct dav_dyn_context | Apache::DAV::DynContext struct dav_dyn_hooks | Apache::DAV::DynHooks struct dav_dyn_module | Apache::DAV::DynModule struct dav_dyn_provider | Apache::DAV::DynProvider struct dav_error | Apache::DAV::Error struct dav_get_props_result | Apache::DAV::PropsResult struct dav_hooks_db | Apache::DAV::HooksDb struct dav_hooks_liveprop | Apache::DAV::HooksLiveprop struct dav_hooks_locks | Apache::DAV::HooksLocks struct dav_hooks_repository | Apache::DAV::HooksRepository struct dav_hooks_vsn | struct dav_if_header | Apache::DAV::IfHeader struct dav_if_state_list | Apache::DAV::StateList ... struct pool | Apache::Pool struct request_rec | Apache:: struct server_rec | Apache::Server ... ``` Defines the mapping from C datatypes to Perl datatypes and classes and tells XSBuilder which datatype are (like) structures ## function map Function map defines the mapping from C functions arguments to Perl arguments - Tell XSBuilder where to place functions and which prefix to strip ``` MODULE=Apache::DAV PACKAGE=guess PREFIX=dav_ ``` - Simple entries in the function map will be mapped 1:1 from C to Perl ``` dav_add_response dav_buffer_append dav_buffer_init dav_buffer_place dav_buffer_place_mem dav_check_bufsize dav_close_propdb dav_collect_liveprop_uris dav_dyn_module_add dav_empty_elem ... ``` - The following map file entry tells XSBuilder that the value of "res\_p" should be returned ``` dav_get_resource | | r, lookup_uri($uri); # get a mod_dav resource object my ($err, $resource) = $subr->get_resource; ``` - You can let XSBuilder insert your custom code, for the interface If you call `"get_props"` from Perl `"glue_dav_get_props"` will be called, which can adjust the arguments and return types as necessary. The actual code for `"glue_dav_get_props"` will be taken from separate include file. ``` dav_get_props | glue_ dav_get_allprops | glue_ ``` xsinclude\\Apache\\DAV\\PropResult\\Apache\_\_DAV\_\_PropResults.h ``` dav_get_props_result * dav_glue_get_props(dav_propdb * db, dav_xml_doc *doc) { dav_get_props_result * result = (dav_get_props_result *)ap_palloc (db -> p, sizeof (dav_get_props_result)) ; *result = dav_get_props(db, doc) ; return result ; } ``` - Arguments can be replaced ``` MODULE=Apache::Array PACKAGE=Apache::Array PREFIX=ap_ ap_make_array | ap_make_array(r->pool, nelts, elt_size) | request_rec *:r, nelts, elt_size ``` ap\_make\_array requires a pool a it's first parameter, we pass the request\_rec from Perl and XSBuilder will take the pool from the request\_rec. ## structure map ``` MALLOC=strdup:$dest = ($type)strdup($src) FREE=strdup:free($src) name ns lang first_cdata following_cdata parent next first_child attr last_child ns_scope propid provider ns_map new MALLOC=strdup:$dest = ($type)ap_pstrdup(obj -> pool, $src) pool elt_size nelts nalloc elts private ``` ### Create a accessor functions for every element and, if requested, a new method ``` $setprop = Apache::DAV::XMLElem -> new ({name => 'prop'}) ; $elem = Apache::DAV::XMLElem -> new ({name => $name, ns => $namespaces}) ; $setprop -> first_child($elem) ; $first = $setprop -> first_child ; ``` ### some datatypes, like strings, requires dynamic allocated memory From \_types.map,which conatins a set of standard types ``` int | IV int * | UNDEFINED unsigned int | UV signed int | IV long | IV long int | IV unsigned long | UV unsigned | UV char * | PV | | | strdup const char * | PV | | | strdup const char ** | UNDEFINED char const * | PV | | | strdup unsigned char * | PV | | | strdup const unsigned char * | PV | | | strdup ... ``` ## callback maps Callback maps have the same options a function maps ``` # dav_hooks_db -> open dav_error *(*)(pool * p,const dav_resource * resource,int ro,dav_db * * pdb) | p, resource, ro=0, has_locks dav_error *(*)(dav_lockdb * lockdb,const dav_resource * resource,int * locks_present) ``` ## Generate the XS files ``` use FindBin ; use lib ($FindBin::Bin) ; require ParseSource ; require WrapXS ; Apache::DAV::WrapXS->run; ``` ### ...and run... ``` C:\perl\msrc\davint>perl xsbuilder\xs_generate.pl Will use mod_dav in C:\perl\msrc\cvs\mod_dav Will use Apache in c:\programme\apache group\apache Parse xsbuilder\maps/_types.map... Parse xsbuilder\maps/dav_type.map... mkdir xs writing...xs//typemap Parse xsbuilder\maps/dav_functions.map... WARNING: Cannot map type int(*)(void * ,const char * ,const char * ) for function ap_table_do WARNING: Cannot map type dav_buffer * for function dav_buffer_append WARNING: Cannot map type dav_buffer * for function dav_buffer_init WARNING: Cannot map type dav_buffer * for function dav_buffer_place WARNING: Cannot map type dav_buffer * for function dav_buffer_place_mem WARNING: Cannot map type dav_buffer * for function dav_check_bufsize WARNING: Cannot map return type int * for function dav_collect_liveprop_uris WARNING: Cannot map type dav_resource * * for function dav_ensure_resource_writable WARNING: Cannot map type dav_buffer * for function dav_lock_get_activelock WARNING: Cannot map type dav_buffer * for function dav_set_bufsize WARNING: Cannot map type int * for function dav_xml2text struct array_header... Parse xsbuilder\maps/dav_structure.map... elt_size... nelts... nalloc... elts... struct dav_buffer... struct dav_datum... dptr... dsize... struct dav_dyn_context... ``` ## Makefile.PL ### We need create a top level Makefile.PL ``` use ExtUtils::MakeMaker (); my $apdir = '/path/to/apache'; my $davdir = '/path/to/moddav'; %MMARGS = ( 'INC' => "-I\"$davdir\" -I\"$apdir/include\" -I\"$apdir/os/unix\" -I\"$dir/xs\" -I\"$dir/xsinclude\"", ) ; open FH, ">xs/mmargs.pl" or die "Cannot open xs/mmargs.pl ($!)" ; print FH Data::Dumper -> Dump ([\%MMARGS], ['MMARGS']) ; close FH ; ExtUtils::MakeMaker::WriteMakefile( 'NAME' => 'Apache::DAV', 'VERSION' => '0.13', %MMARGS, ); ``` ### Makefile.PL's for all class are generated automaticly ``` C:\perl\msrc\davint>perl Makefile.PL Will use Apache in c:\programme\apache group\apache Will use mod_dav in C:\perl\msrc\cvs\mod_dav Checking if your kit is complete... Looks good Writing Makefile for Apache::Array Writing Makefile for Apache::DAV::Datum Writing Makefile for Apache::DAV::DynContext Writing Makefile for Apache::DAV::DynHooks Writing Makefile for Apache::DAV::DynModule Writing Makefile for Apache::DAV::DynProvider Writing Makefile for Apache::DAV::Error Writing Makefile for Apache::DAV::HooksDb Writing Makefile for Apache::DAV::HooksLiveprop Writing Makefile for Apache::DAV::HooksLocks Writing Makefile for Apache::DAV::HooksRepository Writing Makefile for Apache::DAV::IfHeader Writing Makefile for Apache::DAV::Lock Writing Makefile for Apache::DAV::LockDB Writing Makefile for Apache::DAV::LockTokenList Writing Makefile for Apache::DAV::LockupResult Writing Makefile for Apache::DAV::PropCtx Writing Makefile for Apache::DAV::PropsResult Writing Makefile for Apache::DAV::Resource Writing Makefile for Apache::DAV::Response Writing Makefile for Apache::DAV::StateList Writing Makefile for Apache::DAV::Text Writing Makefile for Apache::DAV::TextHeader Writing Makefile for Apache::DAV::WalkerCtx Writing Makefile for Apache::DAV::XMLAttr Writing Makefile for Apache::DAV::XMLDoc Writing Makefile for Apache::DAV::XMLElem Writing Makefile for Apache::DAV Writing Makefile for Apache::TableEntry Writing Makefile for Apache Writing Makefile for WrapXS Writing Makefile for Apache::DAV ``` ### and now compile... ## How does it go on... ### Generating documentation XSBuilder already extracts source comments for functions and structures. It also parses doxygen comments, which are used in Apache 2.0. Lyle Brooks has started on automaticly createing POD files from this information. ### Improving callbacks Callbacks are the main area that needs improvement. ### Bring it back to mod\_perl 2.0 ### First version will be released just after the conference to CPAN ### Any feedback and help appreciated ### Questions?