[Textpow 1.3.1] warning on bad unicode syntax #1

grosser · 2013-08-28T11:26:13Z

Introducing myself : I work in a French company, developing enhancements on our Redmine instance.
I am migrating our developments to the last version of Redmine using Rails 3.

Some Textpow syntax yaml files trigger an annoying warning :
Ignored utf8 regex error invalid multibyte escape: /($)[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*?\b/

I tried to see why, and here is what I found :

The two syntax triggering the warning are :
textpow-1.3.1/lib/textpow/syntax/source.smarty.syntax: match: ($)[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]?\b
textpow-1.3.1/lib/textpow/syntax/source.mel.syntax: match: ($)[a-zA-Z\x7f-\xff][a-zA-Z0-9_\x7f-\xff]_?\b

What I understand is that these regexp contain an unicode range \x7f-\xff, but with rails 3 this unicode range is interpreted as real unicode.
The problem is that the range \x7f-\xff is outside utf8 legal values.

I know that you can specify to not interpret unicode character in full regular expressions, with adding a n at the end :
//no
But the syntax is different in the yam file (no /...../)
I did not find how to specify a range of ascii characters between 127 and 255.

The other question is why smarty and mel want such characters in variables names ?

Have you other ideas about this issue ?

grosser · 2013-08-28T11:38:34Z

replacing them with [^:ascii:] could work on ruby 1.9+ and ree+onigumura not sure if that's exactly the same but should be pretty good...
would also solve the TODO * smart + mel + applescript have some invalid utf8 characters that are currently ignored (search for INVALID_UTF8))

Utopism · 2013-08-28T12:41:19Z

The syntax to replace are :
[a-zA-Z_\x7f-\xff]
meaning :

a $
a second character containing :

a letter uppercase OR
the '_' character OR
a non printable ascii character whose code is in range [127, 255]

[a-zA-Z0-9_\x7f-\xff]
meaning :

0 to n times :

a letter uppercase OR
the '_' character OR
a digit OR
a non printable ascii character whose code is in range [127, 255]

grosser · 2013-08-28T12:50:49Z

aaah thought it meant all of unicode land :D

then not sure how to fix this, if you add the /n and match against a
unicode string kaboom :/

"Ü" =~ /\x7f-\xff/ -> boom

On Wed, Aug 28, 2013 at 2:41 PM, Ennder [email protected] wrote:

The syntax to replace are :

[a-zA-Z_\x7f-\xff]
meaning :

a $

a second character containing :

a letter uppercase OR

the '_' character OR

a non printable ascii character whose code is in range [127, 255]

[a-zA-Z0-9_\x7f-\xff]
meaning :

0 to n times

a letter uppercase OR

the '_' character OR

a digit OR

a non printable ascii character whose code is in range [127, 255]

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23410870
.

Utopism · 2013-08-28T16:08:23Z

We should perhaps find in upstream if ascii characters [127, 255] are really accepted.

grosser · 2013-08-28T16:20:51Z

might as well just kill the extra condition... take a small parsing bug vs
a global problem

On Wed, Aug 28, 2013 at 6:08 PM, Ennder [email protected] wrote:

We should perhaps find in upstream if ascii characters [127, 255] are
really accepted.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23426808
.

Utopism · 2013-08-29T08:36:25Z

You're right, I don't think such variable names cases will happen often !

Utopism · 2014-01-21T10:54:30Z

Hi again,

With Rails 3.2 logs are now filled with these warnings :

/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has '-' without escape: /\b([a-zA-Z-:]+)/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: invalid back reference: /\\(\n\d+|\k\w+|(?<!\|)\g\w+)/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: invalid subexp call: /\\(\n\d+|\k\w+|(?<!\|)\g\w+)/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has '-' without escape: /(\.\.)\s[A-z][A-z0-9-_]+(::)\s*$/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: nested repeat operator ? and * was replaced with '*': /^\\documentclass(?!.*\{beamer\})|^<<(.?*)>>=$/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: nested repeat operator ? and * was replaced with '*': /^<<(.?*)>>=|\\begin\{.*\}/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: nested repeat operator ? and * was replaced with '*': /^@(.?*)$|\\end\{.*\}/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /(?=(\[<)(?![^\[]+?[^>]]))/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /(\[<)(?=.*?>])/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /\[<|>]/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has '-' without escape: /\b([a-zA-Z-_:]+)\s*(=)/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has '-' without escape: /\b([a-zA-Z-_:]+)/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /<|>|]]>|--/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /<|>|]]>|--/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /<|>|]]>|--/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /<|>|]]>|--/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has ']' without escape
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has '-' without escape: /\b([a-zA-Z-:]+)/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /[\[]]+/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /[\[]]+/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: regular expression has ']' without escape: /[\[]]+?/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has ']' without escape: /(\[)([^]]*)(\]) *(\[)(.*?)(\])/
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has ']' without escape
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has ']' without escape
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has ']' without escape
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has ']' without escape
/var/lib/gems/1.9.1/gems/textpow-1.3.1/lib/textpow/syntax.rb:120: warning: character class has ']' without escape

grosser · 2014-01-22T01:07:29Z

hmm maybe print some more info around there and see what syntax/line this is coming from -> PR ?

Utopism · 2014-07-16T16:06:01Z

Hi, here are more informations :
gem ultraviolet V1.0.1
gem textpow V1.3.1

lines that cause warnings in the textpow syntax files :

lib/textpow/syntax/text.tex.latex.sweave.syntax
L8 :
firstLineMatch: ^\documentclass(?!.{beamer})|^<<(.?)>>=$
lib/textpow/syntax/text.tex.latex.sweave.syntax:foldingStartMarker
L11 :
foldingStartMarker: ^<<(.?)>>=|\begin{.}
lib/textpow/syntax/text.html.tt.syntax
lib/textpow/syntax/text.html.mt.syntax
L94 :
match: \b([a-zA-Z-:]+)\s*(=)
Can be fixed by escaping the -
match: \b([a-zA-Z-:]+)\s*(=)

All these cases can be foud with a grep -R "A-Z-" *

lib/textpow/syntax/text.html.strict.active4d.syntax
lib/textpow/syntax/text.html.asp.net.syntax

Some others warnings will be difficult to catch, but this is a beginning

grosser · 2014-07-16T16:41:39Z

can you throw that into a PR ?
I kind of curious, what are you using textpow for ?

Utopism · 2014-07-16T16:45:26Z

lib/textpow/syntax/text.restructuredtext.syntax
Can be fixed by escaping the -
L28 :
match: (..)\s[A-z][A-z0-9-_]+(::)\s*$
lib/textpow/syntax/text.bbcode.syntax
[[]]+ can be replaced by
[[]]+
grep -R '[]]' *

Utopism · 2014-07-16T16:47:17Z

Texpow is used to syntax highlight source files in Redmine (plugin redmine_ultraviolet),

Il try to provide a PR.

grosser · 2014-07-16T17:02:47Z

Feel free to add a link in the readme, always nice to have more real-life
examples :)

On Wed, Jul 16, 2014 at 9:47 AM, Ennder [email protected] wrote:

Texpow is used to syntax highlight source files in Redmine (plugin
redmine_ultraviolet),

Il try to provide a PR.

—
Reply to this email directly or view it on GitHub
#1 (comment).

Utopism · 2014-07-16T17:36:17Z

For the utf8 issue :

Regexp.new
When the lang parameter is n’ orN’ sets the regexp no encoding.

It's what is proposed here :
http://osdir.com/ml/ruby-talk/2009-04/msg00459.html

It seems to pass, don't know if this will lead to a regression

grosser · 2014-07-16T17:38:27Z

if something breaks we'll add more tests, no warnings is a nice win :)

On Wed, Jul 16, 2014 at 10:36 AM, Ennder [email protected] wrote:

For the utf8 issue :

Regexp.new
When the lang parameter is n’ orN’ sets the regexp no encoding.

It's what is proposed here :
http://osdir.com/ml/ruby-talk/2009-04/msg00459.html

It seems to pass, don't know if this will lead to a regression

—
Reply to this email directly or view it on GitHub
#1 (comment).

modifié: lib/textpow/syntax.rb modifié: lib/textpow/syntax/source.camlp4.ocaml.syntax modifié: lib/textpow/syntax/source.ocaml.syntax modifié: lib/textpow/syntax/text.bbcode.syntax modifié: lib/textpow/syntax/text.html.asp.net.syntax modifié: lib/textpow/syntax/text.html.mt.syntax modifié: lib/textpow/syntax/text.html.strict.active4d.syntax modifié: lib/textpow/syntax/text.html.tt.syntax modifié: lib/textpow/syntax/text.html.twiki.syntax modifié: lib/textpow/syntax/text.html.xhtml.1-strict.syntax modifié: lib/textpow/syntax/text.restructuredtext.syntax modifié: lib/textpow/syntax/text.tex.latex.sweave.syntax

Reference to Redmine added modifié: README.rdoc

Utopism · 2014-07-16T17:59:44Z

Hi, I have done a PR, please check it, thanks

Utopism referenced this issue in Smile-SA/textpow Jul 16, 2014

issue #1 [Textpow 1.3.1] warning on bad unicode syntax

6c68b5d

Reference to Redmine added modifié: README.rdoc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Textpow 1.3.1] warning on bad unicode syntax #1

[Textpow 1.3.1] warning on bad unicode syntax #1

grosser commented Aug 28, 2013

grosser commented Aug 28, 2013

Utopism commented Aug 28, 2013

grosser commented Aug 28, 2013

Utopism commented Aug 28, 2013

grosser commented Aug 28, 2013

Utopism commented Aug 29, 2013

Utopism commented Jan 21, 2014

grosser commented Jan 22, 2014

Utopism commented Jul 16, 2014

grosser commented Jul 16, 2014

Utopism commented Jul 16, 2014

Utopism commented Jul 16, 2014

grosser commented Jul 16, 2014

Utopism commented Jul 16, 2014

grosser commented Jul 16, 2014

Utopism commented Jul 16, 2014

[Textpow 1.3.1] warning on bad unicode syntax #1

[Textpow 1.3.1] warning on bad unicode syntax #1

Comments

grosser commented Aug 28, 2013

grosser commented Aug 28, 2013

Utopism commented Aug 28, 2013

grosser commented Aug 28, 2013

Utopism commented Aug 28, 2013

grosser commented Aug 28, 2013

Utopism commented Aug 29, 2013

Utopism commented Jan 21, 2014

grosser commented Jan 22, 2014

Utopism commented Jul 16, 2014

grosser commented Jul 16, 2014

Utopism commented Jul 16, 2014

Utopism commented Jul 16, 2014

grosser commented Jul 16, 2014

Utopism commented Jul 16, 2014

grosser commented Jul 16, 2014

Utopism commented Jul 16, 2014