--- name: "allpairs_master" section: "1" platform: "linux" package: "coop-computing-tools" version: "1:7.14.5-1+b2" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/allpairs_master" --- # allpairs\_master(1) executes All-Pairs workflow in parallel on distributed systems ## NAME **allpairs\_master** - executes All-Pairs workflow in parallel on distributed systems ## SYNOPSIS FCallparis\_master \[options\] *\* *\* *\*FT ## DESCRIPTION **allpairs\_master** computes the Cartesian product of two sets (*\ and \), generating a matrix where each cell* *M\[i,j\] contains the output of the function F (\) on* *objects A\[i\] (an item in \) and B\[j\] (an item in* *\). The resulting matrix is displayed on the standard output,* *one comparison result per line along with the associated X and Y indices.* **allpairs\_master** *uses the Work Queue system to distribute tasks among* *processors. Each processor utilizes the* **[allpairs\_multicore(1)](/allpairs_multicore/1.md)** *program* *to execute the tasks in parallel if multiple cores are present. After starting* **allpairs\_master***, you must start a number of* **[work\_queue\_worker(1)](/work_queue_worker/1.md)** *processes on remote machines. The workers will then connect back to the master* *process and begin executing tasks.* ## OPTIONS - **-p,--port=*\*** *The port that the master will be listening on.* - **-e,--extra-args=*\*** *Extra arguments to pass to the comparison function.* - **-f,--input-file=*\*** *Extra input file needed by the comparison function. (may be given multiple times)* - **-o,--debug-file=*\*** *Write debugging output to this file. By default, debugging is sent to stderr (":stderr"). You may specify logs to be sent to stdout (":stdout") instead.* - **-O,----output-file=*\*** *Write task output to this file (default to standard output)* - **-t,--estimated-time=*\*** *Estimated time to run one comparison. (default chosen at runtime)* - **-x,--width=*\*** *Width of one work unit, in items to compare. (default chosen at runtime)* - **-y,--height=*\*** *Height of one work unit, in items to compare. (default chosen at runtime)* - **-N,--project-name=*\*** *Report the master information to a catalog server with the project name - \* - **-P,--priority=*\*** *Priority. Higher the value, higher the priority.* - **-d,--debug=*\*** *Enable debugging for this subsystem. (Try -d all to start.)* - **-v,--version** *Show program version.* - **-h,--help** *Display this message.* - **-Z,--port-file=*\*** *Select port at random and write it to this file. (default is disabled)* - **--work-queue-preferred-connection=*\*** *Indicate preferred connection. Chose one of by\_ip or by\_hostname. (default is by\_ip)* ## EXIT STATUS *On success, returns zero. On failure, returns non-zero.* ## EXAMPLES *Let's suppose you have a whole lot of files that you want to compare all to* *each other, named FCaFT, FCbFT, FCcFT, and so on. Suppose that you also* *have a program named* **compareit** *that when invoked as FCcompareit a bFT* *will compare files FCaFT and FCbFT and produce some output summarizing the* *difference between the two, like this:* ``` ``` - *a b are 45 percent similar* *To use the allpairs framework, create a file called FCset.listFT that lists each of* *your files, one per line:* ``` ``` - *a*\ *b*\ *c*\ *...* *Because* **allpairs\_master** *utilizes* **[allpairs\_multicore(1)](/allpairs_multicore/1.md)***, so please* *make sure* **[allpairs\_multicore(1)](/allpairs_multicore/1.md)** *is in your PATH before you proceed.To run* *a All-Pairs workflow sequentially, start a single* **[work\_queue\_worker(1)](/work_queue_worker/1.md)** *process in the background. Then, invoke* **allpairs\_master***.* ``` ``` - *% work\_queue\_worker localhost 9123 &*\ *% allpairs\_master set.list set.list compareit* *The framework will carry out all possible comparisons of the objects, and print* *the results one by one (note that the first two columns are X and Y indices in* *the resulting matrix):* ``` ``` - *1 1 a a are 100 percent similar*\ *1 2 a b are 45 percent similar*\ *1 3 a c are 37 percent similar*\ *...* *To speed up the process, run more* **[work\_queue\_worker(1)](/work_queue_worker/1.md)** *processes on* *other machines, or use* **[condor\_submit\_workers(1)](/condor_submit_workers/1.md)** *or* **[uge\_submit\_workers(1)](/uge_submit_workers/1.md)** *to start hundreds of workers in your local batch* *system.* *The following is an example of adding more workers to execute a All-Pairs* *workflow. Suppose your* **allpairs\_master** *is running on a machine named* *barney.nd.edu. If you have access to login to other machines, you could simply* *start worker processes on each one, like this:* ``` ``` - *% work\_queue\_worker barney.nd.edu 9123* *If you have access to a batch system like Condor, you can submit multiple* *workers at once:* ``` ``` - *% condor\_submit\_workers barney.nd.edu 9123 10*\ *Submitting job(s)..........*\ *Logging submit event(s)..........*\ *10 job(s) submitted to cluster 298.* ## COPYRIGHT *The Cooperative Computing Tools are Copyright (C) 2022 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.* ## SEE ALSO - • *The Cooperative Computing Tools (***"http://ccl.cse.nd.edu/software/manuals"***)* - • *All-Pairs User Manual (***"http://ccl.cse.nd.edu/software/manuals/allpairs.html"***)* - • *Work Queue User Manual (***"http://ccl.cse.nd.edu/software/manuals/workqueue.html"***)* - • **[work\_queue\_worker(1)](/work_queue_worker/1.md)** - • **[condor\_submit\_workers(1)](/condor_submit_workers/1.md)** - • **[uge\_submit\_workers(1)](/uge_submit_workers/1.md)** - • **[allpairs\_multicore(1)](/allpairs_multicore/1.md)**