XML Service Common

This page has a few ideas to be talked about at Common, maybe next big improvement on the horizon.

go faster XMLSERVICE

XMLSERVICE was designed to be called from anywhere, using any language, or, in fact just HTML/XML (no language at all). Up front is this design, and, embodied practical outcome trade of ‘convenience and flexibility’ vs. ‘high performance’ custom call ALWAYS faster. We should not violate design ‘convenience and flexibility’, but completely reasonable to explore xmlservice ‘go faster’.

Go faster #1 - stateless and state full available today

*here - stateless
- reloading the ILE *PGM (slow)
*sbmjob - state full 2nd time(+)
- no reload ILE *PGM (fast)
*submit - state full 1st time
- reload program (slow)
<record>
<label>S_PGM_INITEND</label>
<ticks>1000001</ticks>
</record>
<record>
<label>S_PGM_INITEND</label>
<ticks>320</ticks>
</record>	
<record>
<label>S_PGM_INITEND</label>
<ticks>347944</ticks>
</record>
  • Alan - Stateful can theoretically become faster over time if the job is kept open and re-used, but because its start-up time is so long, and people often don’t manage them to be long-running, stateful mode ends up slower. Stateless generally gets my vote. It could be worth a test using *fly in stateful mode, running the test a few times to see if anything improves, but in practice I’ve found people can’t handle stateful mode.
  • Tony - State FULL gets my vote. As you can see, actual program call QSNDDTAQ/QRCVDTAQ is much faster with state full (*submit), over stateless (*here). Basically state full (*sbmjob) avoids all ILE load time (significant time). But, yes, “customer handle state full”, you must allow XTOOLKIT job to live, not kill it with *immed, etc. (no operators killing stuff willy nilly).
  • Unchecked - i did not run db2 persistent test, but it occurs persistent connection used with stateless may have cleansing effect on ILE load of *PGM … BUT … you will have to try it to find out if ticks’ get even close to state full (perception is often wrong rule).

Go faster #2 - What about “pure” PASE implementation?

i am not discouraging a PASE toolkit effort, but blind ‘PASE only’ design may lead into a hole, wherein, we are better off expanding roles of XMLSERVICE.

Example: _PGMCALL to QSNDDTAQ/QRCVDTAQ

  • Simple enough, PASE to ILE, pointers aligned, data in place, _PGMCALL (bingo)
  • task #1: data queue data most often is represented by some “structure” …
    • … maybe some character data?
      • we must add conversion between PASE (ASCII) and data queue (EBCDIC).
        a) PASE conversion operations are iconv, but not all conversion sets are loaded, therefore may introduce requirement on customer to load secondary languages on IBM i
        b) ILE conversion, yes, all tables are loaded base machine, but, now we have multiple calls into ILE, _PGMCALL for QSNDDTAQ/QRCVDTAQ, and more calls for iconv conversion
        c) maybe, we are better off just calling something like XMLSERVICE, then calls user program (strike #1 for custom _PGMCALL)
    • … maybe some pack decimal or zone decimal data?
      a) PASE has no packed decimal or zone decimal construct, in fact, difficult to turn a float/real into a packed with any decimal accuracy
      b) ILE RPG naturally handles packed decimal and zone decimal, but, now we have multiple calls into ILE, _PGMCALL for QSNDDTAQ/QRCVDTAQ, and more calls for packed decimal and zone decimal
      c) maybe, we are better off just calling something like XMLSERVICE, then calls user program (strike #2 for custom _PGMCALL)
    • … in practice, mismatch is common across PASE vs. ILE primitives, and, well, almost always you end up with two choices, wherein, ILE is natural, and PASE struggles (strike #3 for custom _PGMCALL)
  • task #2: do we want this call to work remote clients (Linux, Windows, remote IBM i)?
    • If yes, then we have to deal with some sort of protocol to xfer bytes between client/server
      • Obviously, this example, data queue interface is running the business, seems short sighted to only allow “good solution” running only on IBM i, therefore, we may start to think about a “better” design over simple _PGMCALL, allowing the local/remote to be less limiting.
      • Long story short, we don’t want to invent a new protocol for IBM i (security concern), and, we know already have DB2 connections everywhere, also WRKRDBDIRE works ‘as is’, so, let’s use DB2 stored procedures to transfer the data both local and remote (a lot like xmlservice again).

Go faster #3 - change the XML flowing between XMLSERVICE and language client

Current XMLSERVICE allows any language specific toolkit to remain completely written in the language of the client (php only, ruby only, nodejs only, etc.). Perhaps careful inspection using slightly different XML protocol, may lead to greater return on xmlservice investment. The following technique is already available in XMLSERVICE, proven much faster, but no toolkit uses <records> to date (Alan understands).

      *   - data='records' - data follows in record format
      *      fast "many records" i/o big data (see below)  (1.7.5)
      *      <parm comment='wsopstdcnt'>
      *       <data type='3s0' enddo='wsopstdcnt'/>
      *      </parm>
      *      <parm comment='findMe1'>
      *       <ds var='findMe1' data='records'>
      *        <ds var='dcRec1_t' array='on'>
      *         <ds var='dcRec1_t'>
      *          <data var='dcMyName1' type='10A'/>
      *          <ds var='dcRec2_t'>
      *           <data var='dcMyName2' type='10A'/>
      *           <ds var='dcRec3_t'>
      *            <data var='dcMyName3' type='10A'/>
      *            <ds var='dcRec_t' dim='999' dou='wsopstdcnt'>
      *             <data var='dcMyName' type='10A'/>
      *             <data var='dcMyJob' type='4096A'/>
      *             <data var='dcMyRank' type='10i0'/>
      *             <data var='dcMyPay' type='12p2'/>
      *            </ds>
      *           </ds>
      *          </ds>
      *         </ds>
      *        </ds>
      *       </ds>
      *       <records delimit=':'>:Rgr:B:Ok:nd1:nd1:1:1.1:...:</records>
      *      </parm>
      *      a) <records delimit=':'> simply match in order input
      *         of any complex structure. Output matches
      *         order input (see above)
      *      b) <records  delimit=':'> delimit can be any character
      *         not in your complex records (see above)
      *      c) works with any <parm> or <return>
      *      d) dou/enddo works, but tricky script to design (be careful)

We need more research here to see if this is the ‘good enough’ answer.

Go faster #4 - what about ‘pooling’ xmlservice jobs

XMLSERVICE state full ‘pooling’ is already available to user whim using ctl = "*sbmjob"; ipc = '/tmp/myroutetojob' . In fact, under many conditions state full programming is MUCH faster over stateless default in most toolkits ctl = "*here" .

The trick is to build a scheme of routing fits the task(s) ipc = '/tmp/myroutetojob1'; ipc = '/tmp/myroutetojob2', ... . However, feedback says that state full XMLSERVICE programming is difficult for ‘normal’ programmers. Worse, IBM i operators have a nasty habit of randomly killing XTOOLKIT jobs.

Technically, state full XMLSERVICE use semaphores to sequence multiple client usage of the route to one-at-time ipc = '/tmp/myroutetojob' . Semaphores are performance impacts.

It could be argued that a ‘pooled’ xmlservice connection could be ‘designed’ only one-user-at-a-time ‘running’, in fact, many other toolkits use a supreme secofr daemon job to spawn their children to a task at a time (no semaphore needed). All in all, we have never fully explored the ins/outs of xmlservice pooling, and, may be a very interesting topic for a Common ‘toolkit’ meeting.

  • As teaser … one could imagine xmlservice adopted role pooling extract away from ‘normal’ customers view/understanding, may result in ‘high performance’ in exactly the same way as ipc = '/tmp/myroutetojob1'; ipc = '/tmp/myroutetojob2', ... , but completely ‘painless’ … well … operators still may kill things, but hey, you can’t fix decades of computer ‘right’ dogma.

Go faster #5 - What about custom protocol?

My first reaction is no. However, there may be a case when used in concert with a pooling idea (previous), a bit of custom protocol could offer generous performance improvement.

For example (only example) …

We could invent a mother xmlservice job that simply routed requests to children xmlservice jobs. Essentially, a knock-off of FastCGI (or CGI ILE named activation), but, probably not sockets, although sockets are possible, maybe even good for async requests (nodejs). Instead of sockets, more likely high speed memeory-2-memory (tagged pointer magic dare i whisper the words).

Go faster #6 - What about “mix” PASE implementation and ILE (similar XMLSERVICE)?

Ultimately, a hybrid solution may produce the best possible result, while maintaining classic XMLSERVICE flexibility / adaptability. However, when talk of performance turns to PASE c code, often, ‘toolkit’ complicity for each language seems to go logarithmic.

We need more ‘mixed’ research here.

XMLSERVICE 5250

A generalized topic of 5250 screen scraping, with eye to near zero investment of existing code impact has been tendered by Aaron. Frankly, as i have never used a screen scraper, i may be either best or worst possible comrade for a group 5250 project. In any event, sounds interesting for a side toolkit Common topic meeting.

5250 XML #1 - What is the possibility of XMLSERVICE being used to call an interactive 5250 program?

  • Aaron - “Harvesting” existing business logic without having to first modularize it into callable programs with parameter lists(common problem with IBM i shops). It would be a macro of some sort, described via XML, that would emulate a user interaction.  For example:
    <macro_list>
     <call  lib="PRD" pgm="CUSTMAINT" />
      <step>
        <occupy fld="CUSNBR" value="1123" />
      </step>
      <step>
        <action key="ENTER" />
      </step>
    </macro_list>
    
  • Alan - OK! It’s a cool idea as long as we enumerate the situations where it’s useful (not a silver bullet to GUI-ize all screens). Aaron’s RPG II example, could be one. I guess any time the source code is either unavailable or un-maintainable would be a solid use case.