Spider for crawling urls

combination of multiple steps to be applied in parellel to data

spider(name = "", queue = list(), steps = list(), pipeline = NA)

Arguments

name

the name of the spider to be used for logging purposes

queue

the list() of urls or initial paths to be used by steps()

steps

a list() of steps() to be applied to the data sequencially

pipeline

the default pipeline() to apply to the data after all steps are applied

Examples

s_multiply <- spider('multiply', queue = 1:4, steps = list(parser( ~ .x * 2)))
s_multiply
#> # A spider: multiply 
#> ### Queue: 4 item(s)
#> ### Steps: 1
#> ( 1 ) A parser: ~.x * 2 

run(s_multiply)
#> Executing parser  ~.x * 2 
#> [[1]]
#> [1] 2
#> 
#> [[2]]
#> [1] 4
#> 
#> [[3]]
#> [1] 6
#> 
#> [[4]]
#> [1] 8
#>