Questions Regarding Combinators, Efficiency And Workflow

[FlipperPlz]

Introduction

Hey again!! The title may be incorrect here but I will try to explain myself as much as possible. Currently as stated in the previous post, I am implementing a context-less preprocessor (context-less as in it can be run on any document starting at any offset) in doing so I have been stumped implementing macros, as you will see I provided a "minimal" example of what I have written so far but I have a few issues so far. In fear of making you think that I'm just sending you a block of code and asking you to "help me" or "fix it" I will try as best as possible to form my issues into proper generalized questions regarding the use of the library, bare with me if you can. :3

Questions

• What would be the most efficient way to have a parser that can optionally be delimited and would it be beneficial to create a custom combinator for such a scenario e.g

//maybe_delimited<T> ::= '(' <T> ')' | <T>;
let string = text::ascii::ident().maybe_delimited(
    just('(').ignored(),
    just(')').ignored(),
); 

• Is there a way to parse a complex type that sort of defines its own syntax as it goes? For example if I wanted to make a parser for a string like type that is prefixed by the number of chars in the string how would I go about this? (e.g given "4fourfive" it would parse "four"), currently all I can think of doing is running map and creating a parser inside of the map closure but then the result ends up being a parser that outputs a parser.

let string = text::int(10).map(| number: &str | {
    any().repeated().exactly(
        number.parse::<usize>().unwrap_or(0)
    )
})

• Finally is there some way to pass in an argument through the parsing process that can be used during mapping?

struct Test(i32);
fn test<'ast>(
    test: &mut Test, 
) -> impl Parser<'ast, &'ast str, &'ast str, extra::Err<Rich<'ast, char>>> {
    any().map( | s | {
        test.0 += 1;
        s
    }
}

If you could answer any of these questions or nudge me in the right direction it would be greatly appreciated. Thank you for making this library and still supporting it

"Minimal" Reproducible Example

This example contains every issue I was running into

use std::collections::HashMap;
use chumsky::prelude::{choice, just, none_of, recursive, Parser, text, extra, Rich};

const STRING_DELIMITER: char = '"';
const PARAM_SEPARATOR: char = ',';
const PARAM_WILDCARD: &'static str = "(),";
const PARAMS_START: char = '(';
const PARAMS_STOP: char = ')';

enum Token<'ast> {
    Text(&'ast str),
    MacroCall(MacroCall<'ast>, String)
}

pub struct MacroCall<'ast> {
    target: &'ast Macro,
    arguments: Option<Vec<String>>
}

pub struct Macro {
    argument_names: Vec<String>,
    value: String,
    scope: u32
}

fn quoted_text<'ast>() -> impl Parser<'ast, &'ast str, &'ast str, extra::Err<Rich<'ast, char>>> {
    none_of(STRING_DELIMITER).repeated().delimited_by(
        just(STRING_DELIMITER).ignored(), just(STRING_DELIMITER).ignored()
    ).to_slice()
}


fn expand_macro(macro_call: &MacroCall, arguments: Option<&Vec<String>>) -> String {
    let expanded = macro_call.target.value.clone();

    match arguments {
        None => expanded,
        Some(args) => {
            let mut expanded = expanded;
            macro_call.target.argument_names.iter()
                .zip(args)
                .for_each(| (name, value) | {
                    expanded = expanded.replace(name, &*value)
                });

            expanded
        }
    }
}

fn text_or_macro<'ast>(
    macro_table: &mut HashMap<String, Macro>, //mutable access is needed outside of this example
) -> impl Parser<'ast, &'ast str, Token<'ast>, extra::Err<Rich<'ast, char>>> {
    recursive(| text_or_macro | {
        text::ascii::ident()
            .map(| identity | {
                //Question 3: We cannot access `macro_table` from within the `Fn` closure
                let Some(candidate_macro) = macro_table.get(identity) else {
                    return Token::Text(identity);
                };
                let argument_count = candidate_macro.arguments();

                if argument_count == 0 {
                    let expanded = expand_macro(candidate_macro, None);
                    let call = MacroCall {
                        target: &candidate_macro,
                        arguments: None,
                    };
                    return Token::MacroCall(call, expanded)
                }
                // Question 1: macro_argument can also be surrounded in parenthesis 
                let macro_argument = choice((
                    quoted_text().map(| text | text.to_string()),
                    text_or_macro.map(| s | match s {
                        Token::Text(s) => s.to_string(),
                        Token::MacroCall(_, s) => s
                    }),
                )).repeated().collect::<Vec<_>>().map(| f | f.join("") );


                macro_argument // Question 2: returns `Parser<I,O,E>` instead of `O`
                    .separated_by(just(PARAM_SEPARATOR).ignored())
                    .exactly(argument_count)
                    .collect()
                    .delimited_by(
                        just('(').ignored(),
                        just(')').ignored()
                    )
                    .map(| arguments| {
                        let expanded = expand_macro(candidate_macro, Some(&arguments));
                        let call = MacroCall {
                            target: &candidate_macro,
                            arguments: Some(arguments),
                        };

                        return Token::MacroCall(call, expanded)
                    });
            })
    })
}```

[FlipperPlz]

semi answer for question 3:

The state and context can be used to hold information for parsing. However with this I don't know how to define context, currently I use the state in map_with to identify the macro written, if one is written I want the argument count to be passed down so later on it can be enforced but I'm currently having trouble trying to figure out how. My first idea was to use an added Context but the Context cant be edited from within map_with so currently I'm at a loss

text::ascii::ident::<ParserInput<&'ast str>, char, Full<Rich<'ast, char, BSpan>, State, ()>>()
    .map_with(| identity, extra| {
        let state: &mut State = extra.state();

        let Some(candidate_macro): Option<&RvMacroDefinition> = state.next_macro(identity) else {
            return Token::Text(identity);
        };

        if !candidate_macro.in_scope() {
            return Token::Text(identity);
        }

        let argument_count = candidate_macro.arguments();
        let call = MacroCall {
            macro_name: identity,
            arguments: None,
        };

        if argument_count == 0 {
            let expanded = candidate_macro.expand(None);
            return Token::MacroCall(call, expanded)
        }

        return Token::MacroCall(call, String::new())
    }).then(
        //E.G here I want to enforce the argument_count identified in the above map
        any().repeated().separated_by(just(Self::PARAM_SEPARATOR).ignored())
            .exactly(argument_count)
    )