Okay, I don’t like the array of tokens, so before adding new features, I’m going to try and change it.

Basically, the kw_ functions should just be able to call nextToken on their received parameters, and continue doing that until there are no more tokens.

This means that I first need to add an END token:

typedef enum
{
  WHITESPACE,
  STRING,
  IDENTIFIER,
  EQUALS,
  END
} TOKEN_TYPE;

And let me also add a constant token for that (like the equal token)

Token TOKEN_end = {.type = END, .value = "-end of tokens-"};

Then, in nextToken, before checking for whitespace, I’ll check for the end of the string

  if (c == '\\0')
  {
    token = &TOKEN_end; // use global end token
  }
  else if (c == ' ' || c == '\\t') // whitespace
  {
   ...

While I’m at it, I think I’ll add a nextTokenIgnoreWhitespace, that returns the next token that isn’t whitespace, meaning it skips over possible whitespace tokens …

Token *nextTokenIgnoreWhitespace(char **text_ptr)
{
  Token *token;

  // skip whitespace
  do
  {
    token = nextToken(text_ptr);
  } while (token->type == WHITESPACE);

  // if the token is not a whitespace, return it
  return token;
}

Since we now always have a TOKEN_end as the last token, this function does not need to test if there are more text available, if the text ends, it will return a TOKEN_end

Now let me rewrite the kw_ functions.

kw_print

I’ll get rid of the for loop, and instead get a token directly - and while that isn’t a END print it, and get the next one. I’ll only print <string> and <identifier>, and ignore anything else.

Token *token = nextToken(&parm);
  while(token->type != END)
  {
    if (token->type == STRING)
    {
      // print value
      printf("%s", token->value);
    }
    else if (token->type == IDENTIFIER)
    {
      // find variable with that name - read it and print the value
      printf("%s", variable_read(token->value));
    }
    // other tokens are just ignored for now - meaning not printed at all
    
    token = nextToken(&parm);
  }  

and it works!

It even works when I just write PRINT without any parameters - no crashes!

kw_let

The LET command was the first one that caused me to think about changing the tokenizer this way, so let’s see if it improves!

The only change will be to replace all the