How to do Pipelines?

[mielvds]

The website says

Pipelines

Use SPARQL Results Sets as input for parametric queries. Decompose your data integration workflow

That sounds interesting! But what do you mean exactly and how does it work? I currently use Apache Airflow to organise a decomposed mapping process, so anything in that direction is helpful.

[enridaga]

About the parametric queries, this is achieved by the CLI, see options -v, -i (the Results Sets as CSV), and -p (use parameters in output file names.

Maybe the best example is in this showcase project, for example:

fx -q imma-artist.sparql -i imma-artists.xml -p "artists/?artistNickname.jsonld" -f json

Where imma-artists.xml is a SPARQL result set.

[mielvds]

aha I see! Well too bad you can't do fx -q imma-artists.sparql -o imma-artists.xml -f xml | fx -q imma-artist.sparql - -p "artists/?artistNickname.jsonld" -f json :)

What I also find useful is splitting mapping queries in multiple queries, you know, for the conginitive load. Can you repeat -i to run multiple queries in sequence on the same data ?

[enridaga]

aha I see! Well too bad you can't do fx -q imma-artists.sparql -o imma-artists.xml -f xml | fx -q imma-artist.sparql - -p "artists/?artistNickname.jsonld" -f json :)

I haven't thought of getting STDIN, not impossible, but it would probably look more like:

fx -q imma-artists.sparql -f xml | fx -q imma-artist.sparql -i -p "artists/?artistNickname.jsonld" -f json` -- where -i without argument would attempt to catch STDIN...

What I also find useful is splitting mapping queries in multiple queries, you know, for the conginitive load. Can you repeat -i to run multiple queries in sequence on the same data ?

Not sure I get this one, you mean repeating -q to execute multiple queries? You can't at the moment, but maybe you want also to treat the output differently, not sure if it's not easier to just write different commands after all.

BTW, if you are interested into pipelines, another way is to nest service clauses, for example:

https://gist.github.com/enridaga/ca5220c1cff14baee9734e61a6676065

I know, if all of that would be well-documented ... but we will get there ;)

[mielvds]

aha I see! Well too bad you can't do fx -q imma-artists.sparql -o imma-artists.xml -f xml | fx -q imma-artist.sparql - -p "artists/?artistNickname.jsonld" -f json :)

I haven't thought of getting STDIN, not impossible, but it would probably look more like:

fx -q imma-artists.sparql -f xml | fx -q imma-artist.sparql -i -p "artists/?artistNickname.jsonld" -f json` -- where -i without argument would attempt to catch STDIN...

Ah yes that why I put the - there because it seems to be a common pattern for stdin, but empty -i also makes sense.

What I also find useful is splitting mapping queries in multiple queries, you know, for the conginitive load. Can you repeat -i to run multiple queries in sequence on the same data ?

Not sure I get this one, you mean repeating -q to execute multiple queries? You can't at the moment, but maybe you want also to treat the output differently, not sure if it's not easier to just write different commands after all.

sorry, yes, I mean -q. It would be slightly more efficient than writing different commands, but indeed, the overall gain would be minimal.

BTW, if you are interested into pipelines, another way is to nest service clauses, for example:

https://gist.github.com/enridaga/ca5220c1cff14baee9734e61a6676065

That's a pretty neat pattern actually! We have a lot of json data stored in RDB records (don't judge us), so if SA were to support RDB access, it would be quite smooth way to handle such a scenario!

(BTW, any plans for JDBC? It's a pain in Java for sure, but I can maybe squeeze out a PR) nvm, saw #6 and looks complicated :)

I know, if all of that would be well-documented ... but we will get there ;)

[justin2004]

you could do something like the unix pipe by running insert queries instead of selects/constructs.

e.g.

curl --silent 'http://localhost:3000/sparql.anything'  \
--header "Accept: text/csv" \
--data-urlencode 'update=
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
PREFIX ex: <http://example.com/>
insert { graph ex:apple { ?s ?p ?o}}
WHERE {
 SERVICE <x-sparql-anything:> {
   fx:properties fx:location "https://www.google.com/search?q=friends" ;
                 fx:media-type "text/html" ;
                 fx:html.browser "firefox" .
   ?s ?p ?o .
 }
}'

then more inserts then eventually a select or construct...

curl --silent 'http://localhost:3000/sparql.anything'  \
--header "Accept: text/csv" \
--data-urlencode 'query=
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
PREFIX ex: <http://example.com/>
select *
WHERE {
   graph ex:apple {?s ?p ?o }
}'

you can think of the pipeline as just incrementally building named graphs.