Fix bugs in Peekable and Flatten when using non-fused iterators

[timvermeulen]

I fixed a couple of bugs with regard to the Peekable and Flatten/FlatMap iterators when the underlying iterator isn't fused. For testing, I also added a NonFused iterator wrapper that panics when next or next_back is called on an iterator that has returned None before, which will hopefully make it easier to spot these mistakes in the future.

Peekable

Peekable::next_back was implemented as

self.iter.next_back().or_else(|| self.peeked.take().and_then(|x| x))

which is incorrect because when the peeked field is Some(None), then None has already been returned from the inner iterator and what it returns from next_back can no longer be relied upon. test_peekable_non_fused tests this.

Flatten

When a FlattenCompat instance only has a backiter remaining (i.e. self.frontiter is None and self.iter is empty), then next will call self.iter.next() every time, so the iter field needs to be fused. I fixed it by giving it the type Fuse<I> instead of I, I think this is the only way to fix it. test_flatten_non_fused_outer tests this.

Furthermore, previously FlattenCompat::next did not set self.frontiter to None after it returned None, which is incorrect when the inner iterator type isn't fused. I just delegated it to try_fold because that already handles it correctly. test_flatten_non_fused_inner tests this.

r? @scottmcm

[rust-highfive]

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-02-10T12:33:13.8508919Z ========================== Starting Command Output ===========================
2020-02-10T12:33:13.8511553Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/aa165ab7-07ad-4243-8557-f7e311981708.sh
2020-02-10T12:33:13.8511794Z 
2020-02-10T12:33:13.8517054Z ##[section]Finishing: Disable git automatic line ending conversion
2020-02-10T12:33:13.8525396Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68915/merge to s
2020-02-10T12:33:13.8527003Z Task         : Get sources
2020-02-10T12:33:13.8527039Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-10T12:33:13.8527121Z Version      : 1.0.0
2020-02-10T12:33:13.8527155Z Author       : Microsoft
---
2020-02-10T12:33:18.0376313Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-02-10T12:33:18.9307459Z ##[command]git config gc.auto 0
2020-02-10T12:33:18.9312936Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-02-10T12:33:18.9316211Z ##[command]git config --get-all http.proxy
2020-02-10T12:33:18.9325617Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/68915/merge:refs/remotes/pull/68915/merge
---
2020-02-10T12:39:36.4728816Z    Compiling serde_json v1.0.40
2020-02-10T12:39:38.1407656Z    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
2020-02-10T12:39:48.5632803Z     Finished release [optimized] target(s) in 1m 34s
2020-02-10T12:39:48.5743153Z tidy check
2020-02-10T12:39:49.5921458Z tidy error: /checkout/src/libcore/tests/iter.rs: too many lines (3012) (add `// ignore-tidy-filelength` to the file to suppress this error)
2020-02-10T12:39:51.2949138Z Found 487 error codes
2020-02-10T12:39:51.2949948Z Found 0 error codes with no tests
2020-02-10T12:39:51.2950280Z Done!
2020-02-10T12:39:51.2950386Z some tidy checks failed
2020-02-10T12:39:51.2950386Z some tidy checks failed
2020-02-10T12:39:51.2954264Z 
2020-02-10T12:39:51.2954571Z 
2020-02-10T12:39:51.2962046Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor"
2020-02-10T12:39:51.2966162Z 
2020-02-10T12:39:51.2966330Z 
2020-02-10T12:39:51.3015501Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
2020-02-10T12:39:51.3015630Z Build completed unsuccessfully in 0:01:44
2020-02-10T12:39:51.3015630Z Build completed unsuccessfully in 0:01:44
2020-02-10T12:39:51.3022491Z == clock drift check ==
2020-02-10T12:39:51.3033675Z   local time: Mon Feb 10 12:39:51 UTC 2020
2020-02-10T12:39:51.8509186Z   network time: Mon, 10 Feb 2020 12:39:51 GMT
2020-02-10T12:39:51.8509275Z == end clock drift check ==
2020-02-10T12:39:52.6497893Z 
2020-02-10T12:39:52.6625090Z ##[error]Bash exited with code '1'.
2020-02-10T12:39:52.6641289Z ##[section]Finishing: Run build
2020-02-10T12:39:52.6659781Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68915/merge to s
2020-02-10T12:39:52.6661912Z Task         : Get sources
2020-02-10T12:39:52.6661959Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-10T12:39:52.6662024Z Version      : 1.0.0
2020-02-10T12:39:52.6662066Z Author       : Microsoft
2020-02-10T12:39:52.6662066Z Author       : Microsoft
2020-02-10T12:39:52.6662112Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-02-10T12:39:52.6662181Z ==============================================================================
2020-02-10T12:39:53.0909843Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-02-10T12:39:53.0990401Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/68915/merge to s
2020-02-10T12:39:53.1111515Z Cleaning up task key
2020-02-10T12:39:53.1112318Z Start cleaning up orphan processes.
2020-02-10T12:39:53.1232744Z Terminate orphan process: pid (3790) (python)
2020-02-10T12:39:53.1694456Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[Dylan-DPC-zz]

r? @Amanieu

[Amanieu]

Furthermore, previously FlattenCompat::next did not set self.frontiter to None after it returned None, which is incorrect when the inner iterator type isn't fused. I just delegated it to try_fold because that already handles it correctly. test_flatten_non_fused_inner tests this.

I am concerned that the generated code is now substantially more complicated. Wouldn't it have been better to simply set frontiter to None after the if?

All the other changes look fine.

[JohnCSimon]

Ping from triage:
@timvermeulen can you please address the comments from Amanieu? Thank you.

[timvermeulen]

Sorry, yep, I'll address them soon.

[timvermeulen]

@Amanieu Thanks for taking a look! I assumed that the generated code for next / next_back would be the same, but benchmarks show that this was indeed a regression, so I've fixed it in the way you suggested.

[Amanieu]

@bors r+

[bors]

📌 Commit 8cf33b0 has been approved by Amanieu

[bors]

⌛ Testing commit 8cf33b0 with merge d939f70...

[bors]

☀️ Test successful - checks-azure
Approved by: Amanieu
Pushing d939f70 to master...

[rust-highfive]

📣 Toolstate changed by #68915!

Tested on commit d939f70.
Direct link to PR: #68915

💔 rls on linux: test-pass → test-fail (cc @Xanewok).